Disaster Recovery

In case of a disaster scenario, you may lose the LDS node. In case you have a backup, you can restore the node from it. In case you do not have a backup, then you will have to rebuild a new LDS node.

This section will give you an overview of the scenarios and help you rebuilding the LDS node.

Assumptions:

  • We are not covering a specific tool which is used to backup the servers or create images of it.

  • The backup you have is a complete backup of the server like VM snapshot.

Restoring the node from backup

  • The LDS node is setup in any of the following ways and it crashes:

    • Setup on a dedicated node.

    • Setup along with MTA, Proxy or Mailbox node.

  • You have a backup of the node.

    • We will assume that you have a full backup of the system (E.g. VM snapshot)

    • The IP address and hostname remains same and there is no change.

Impact

  • The server will go in Grace Period after approx 5-10 minutes.

  • Restricted account operations.

  • If the dedicated LDS node crashed, then the setup will run in Grace Period.

  • Along with LDS, if any other service was running on the node, then that functionality will not be available until we restore the node. E.g. if MTA service was running, then the email operations will be impacted.

Functionality in Grace Period

Following functionality is available when the system is in Grace Period:

  • Email Operations - All the email operations will continue to function and there will be no impact or interruption for the end users.

  • Account modification - Users can modify any of the settings for their account. E.g. creating signatures or filters, changing password, etc.

  • Network features - All the network features will continue to function except restoring of an account.

Following functionality is not available when the system is in Grace Period:

  • Account operations:

    • You cannot create a new user or delete an existing user.

    • You cannot modify/update the following features for the existing users - EWS, SMIME, ActiveSync & ZCO

    • You cannot restore the account from a backup.

Verify the license status

  • You will need to check the license status and verify the server has entered into the grace period.

As a zimbra user, execute the following command on the mailbox node:

$ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is in grace period

Restoring the node

Zimbra Daffodil (v10.1) onwards, the licenses are linked with the device id of the server. If after the restore, the device ID is changed, please contact support team to remove the linked device id for your license.

In an ideal case, the device id will change when you are changing the LDS server when restoring the node or creating a new LDS on a new server.

Following are the steps:

  1. Restore the node from the backup.

    1. In case the node is not a dedicated LDS node, verify the expected Zimbra services are up and running.

  2. Validate the hostname and IP address.

  3. Verify the device id of the node is same as registered for the license. Execute the following commands as zimbra user:

    1. If the device id is same, then no error will be displayed:

      $ zmlicense -p
Current activated license : 539165225482003729
Activated Product Version : 10.1.0_GA_4633
LDS Device Id : *KgQI7MUZS0jU2kKo3VBk*

Feature : AccountsLimit
	Status: Authorized for use
	.
	.
	.

    2. If the device id is not same, then the following error will get displayed:

      $ zmlicense -p
2024-07-01 07:02:41,111 [main] ERROR: license cache -> Couldn't fetch license details
com.zimbra.common.service.ServiceException: system failure: Error while getting license status: 539463447635898378, msg:
	at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:296) ~[zimbracommon.jar:10.1.0_GA_4633]
    .
    .
    .
	... 7 more
system failure: license cache -> Couldn't fetch license status

    3. Contact the support team to get the device id deleted for the license. Once the device id is deleted, the LDS will automatically assign the new device id and the mailbox servers will get connected to the LDS.

  4. If the device id is same, the services will be started.

  5. Verify the LDS service is running. As a zimbra user, execute the following command:

    $ zmlicensectl --service status
license daemon service is running.

  6. In case the service is not running, restart the LDS service. As a zimbra user, execute the following command:

    $ zmlicensectl --service restart

  7. The mailbox servers should get auto connected to the LDS successfully.

  8. The license should exit the grace period and run in normal mode.

  9. Verify the license status. As a zimbra user, execute the following command:

    $ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is OK

  10. You should now be able to perform all the Zimbra operations.

Rebuilding the node (No backup)

  • The LDS node is setup in any of the following ways and it crashes:

    • Setup on a dedicated node.

    • Setup along with MTA, Proxy or Mailbox node.

  • You have NO backup of the node, or the backup is not usable.

  • You have to rebuild the node from scratch and assign new IP address/hostname.

Impact

  • The server will go in Grace Period after approx 5-10 minutes.

  • Restricted account operations.

  • If the dedicated LDS node crashed, then the setup will run in Grace Period.

  • Along with LDS, if any other service was running on the node, that functionality will not be available until we restore the node. E.g. if MTA service was running, then the email operations will be impacted.

Functionality in Grace Period

Following functionality is available when the system is in Grace Period:

  • Email Operations - All the email operations will continue to function and there will be no impact or interruption for the end users.

  • Account modification - Users can modify any of the settings for their account. E.g. creating signatures or filters, changing password, etc.

  • Network features - All the network features will continue to function except restoring of an account.

Following functionality is not available when the system is in Grace Period:

  • Account operations:

    • You cannot create a new user or delete an existing user.

    • You cannot modify/update the following features for the existing users - EWS, SMIME, ActiveSync & ZCO

    • You cannot restore the account from a backup.

Verify the license status

  • You will need to check the license status and verify the server has entered into the grace period.

As a zimbra user, execute the following command on the mailbox node:

$ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is in grace period

Rebuilding the node

During the installation, choose the packages as per the services you are rebuilding.

Following are the steps:

  1. Remove the older LDS details from mailbox configuration. As a zimbra user, execute the following command on mailbox node:

    zmprov mcf zimbraLicenseDaemonServerHost ""

  2. To update the cache and remove the LDS from it, execute the command as zimbra user:

    zmprov fc all

  3. Contact the support team to delete the device id for the license. In the meantime, you can proceed with further steps.

  4. Execute ./install.sh.

  5. In case any other Zimbra services were running on the node, select the required packages.

  6. Type y and press Enter to install the zimbra-license-daemon package.

    Install zimbra-license-daemon [Y]

  7. Complete the rest of the installation.

  8. Verify the LDS service is running. As a zimbra user, execute the following command:

    $ zmlicensectl --service status
license daemon service is running.

  9. In case the service is not running, restart the LDS service. As a zimbra user, execute the following command:

    $ zmlicensectl --service restart

  10. The node is now up and running.

  11. Verify the zimbraLicenseDaemonServerHost is set to the new LDS node. Execute the command as zimbra user:

    zmprov gcf zimbraLicenseDaemonServerHost
zimbraLicenseDaemonServerHost: <new_lds_hostname>

  12. The mailbox servers should get auto connected to the LDS successfully.

  13. The licese should exit the grace period and run in normal mode.

  14. Verify the license status. As a zimbra user, execute the following command:

    $ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is OK

  15. You should now be able to perform all the Zimbra operations.

Setting up a standby LDS node

You can also setup a standby LDS node. In case of a disaster scenario, you can switch to the standby node.

Installing standby LDS node

When installing a parallel LDS node, the installer will identify that you already have an existing LDS node and will give a warning. We will have to manually install the LDS packages after the installation is complete.

Following are the detailed steps:

  1. Execute ./install.sh.

  2. Type y and press Enter to install the zimbra-license-daemon package.

    Install zimbra-license-daemon [Y]

  3. Following warning will get displayed:

    Configuring license daemon service...
WARNING: license-daemon service already installed on <existing_lds_node_hostname>
Removing zimbra-license-daemon from this host...done.

  4. Ignore the warning and complete the rest of the installation.

  5. Install the LDS packages

    1. For Ubuntu, execute the command as root user:

      apt-get install zimbra-license-daemon

    2. For RHEL, execute the command as root user:

      yum install zimbra-license-daemon

  6. Add license-daemon service to zimbra service list. As a zimbra user, execute the following command:

    zmprov ms `zmhostname` +zimbraServiceEnabled license-daemon

  7. Verify the LDS service is running. As a zimbra user, execute the following command:

    $ zmlicensectl --service status
license daemon service is running.

  8. In case the service is not running, restart the LDS service. As a zimbra user, execute the following command:

    $ zmlicensectl --service restart

  9. Restart zmconfigdctl. As a zimbra user, execute the following command:

    $ zmconfigdctl restart

  10. The node is now setup as a standby node.

Performing a switchover to the standby node

In case of a DR scenario, following are the steps to switchover to the standby LDS node

  1. Check the license status and verify the server has entered into the grace period. As a zimbra user, execute the following command on the mailbox node:

    $ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is in grace period

  2. Contact the Zimbra Support team to get the device id deleted for the license. Once the device id is deleted, the dedicated LDS will automatically assign the new device id and the mailbox servers will get connected to the dedicated LDS.

  3. Check the existing LDS host configuration. It should display the existing LDS node. As a zimbra user, execute the following command:

    zmprov gcf zimbraLicenseDaemonServerHost
zimbraLicenseDaemonServerHost: <existing_lds_node_hostname>

  4. On a mailbox node, execute the command as a zimbra user:

    zmprov mcf zimbraLicenseDaemonServerHost <standby_lds_node_hostname>

  5. Verify the configuration has been updated:

    zmprov gcf zimbraLicenseDaemonServerHost
zimbraLicenseDaemonServerHost: <standby_lds_node_hostname>

  6. The licese should exit the grace period and run in normal mode.

  7. Verify the license status. As a zimbra user, execute the following command:

    $ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is OK

  8. You should now be able to perform all the Zimbra operations.

Moving an LDS node to a dedicated LDS node

If you want to move the LDS service from an exsting node to a dedicated node, you can do that with minimal downtime of the LDS node.

Setup:

  1. LDAP node.

  2. MTA+Proxy node.

  3. Mailbox node1 + LDS.

  4. Mailbox node2.

The LDS is setup on Mailbox node1. You want to move the LDS from Mailbox node1 to a dedicated LDS node.

Installing LDS on a dedicated node

When installing a parallel LDS node, the installer will identify that you already have an existing LDS node and will give a warning. We will have to manually install the LDS packages after the installation is complete.

Following are the detailed steps:

  1. Execute ./install.sh.

  2. Type y and press Enter to install the zimbra-license-daemon package.

    Install zimbra-license-daemon [Y]

  3. Following warning will get displayed:

    Configuring license daemon service...
WARNING: license-daemon service already installed on <existing_lds_node_hostname>
Removing zimbra-license-daemon from this host...done.

  4. Ignore the warning and complete the rest of the installation.

  5. Install the LDS packages

    1. For Ubuntu, execute the command as root user:

      apt-get install zimbra-license-daemon

    2. For RHEL, execute the command as root user:

      yum install zimbra-license-daemon

  6. Add license-daemon service to zimbra service list. As a zimbra user, execute the following command:

    zmprov ms `zmhostname` +zimbraServiceEnabled license-daemon

  7. Verify the LDS service is running. As a zimbra user, execute the following command:

    $ zmlicensectl --service status
license daemon service is running.

  8. In case the service is not running, restart the LDS service. As a zimbra user, execute the following command:

    $ zmlicensectl --service restart

  9. Restart zmconfigdctl. As a zimbra user, execute the following command:

    $ zmconfigdctl restart

  10. The node is now setup as a dedicated LDS node.

Performing a switchover to the dedicated node

The setup will run in Grace Period till the switchover is complete to the dedicated LDS node and the mailbox nodes are connected to it.

Following are the steps to switchover to the dedicated LDS node

  1. Stop the LDS service on the Mailbox Node1. As a zimbra user, execute the following command:

    $ zmlicensectl --service stop

  2. Remove the LDS packages on Mailbox Node1:

    1. For Ubuntu, execute the command as root user:

      apt-get purge zimbra-license-daemon

    2. For RHEL, execute the command as root user:

      yum remove zimbra-license-daemon

  3. Contact the Zimbra Support team to get the device id deleted for the license. Once the device id is deleted, the dedicated LDS will automatically assign the new device id and the mailbox servers will get connected to the dedicated LDS.

  4. Check the license status and verify the server has entered into the grace period. As a zimbra user, execute the following command on the mailbox node:

    $ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is in grace period

  5. Check the existing LDS host configuration. It should display the existing LDS node. As a zimbra user, execute the following command:

    zmprov gcf zimbraLicenseDaemonServerHost
zimbraLicenseDaemonServerHost: <existing_lds_node_hostname>

  6. On a mailbox node, execute the command as a zimbra user:

    zmprov mcf zimbraLicenseDaemonServerHost <dedicated_lds_node_hostname>

  7. Verify the configuration has been updated:

    zmprov gcf zimbraLicenseDaemonServerHost
zimbraLicenseDaemonServerHost: <dedicated_lds_node_hostname>

  8. The licese should exit the grace period and run in normal mode.

  9. Verify the license status. As a zimbra user, execute the following command:

    $ zmlicense -c
checking license status. . .
Current license code : 539165225482003729, Activation status: License is OK

  10. You should now be able to perform all the Zimbra operations.