VMware Stack Upgrade

I’m working on a plan to upgrade our existing VMware Stack and wanted to write a detailed post about all components involved and the order of upgrade.

Current State –

·        Primary and Recovery site with XtremIO Arrays on both ends with physical RecoverPoint appliances for array based replication.

·        vCenter Servers on both sites (with embedded Platform Services Controller) running on Windows Server 2012 R2 with an external SQL Database.

·         ESXi 6.0.0, 3825889

·         Site Recovery Manager (SRM) 6.1.1.13825

·         Storage Replication Adapter (SRA) 2.2.0.3

·         XtremIO (XIOS) 4.0.15-24, XMS 4.2.1

·         RecoverPoint (RPA) 4.4.1

·         Avamar 7.3

Future State –

·         vCenter Server Appliance (VCSA) 6.5 U1f

·         ESXi 6.5, 7388607

·         SRM 6.5.1, 6014840

·         SRA 2.2.0.3

·         XIOS 6.0.1, XMS 6.0.1

·         RecoverPoint (RPA) 5.1

·         Avamar 7.5

 There are numerous inter-dependencies to get to the future state.

·         Avamar 7.3 is not compatible with vCenter 6.5

·         RPA 4.4.1 is not compatible with ESXi 6.5

·         Upgrading SRM from 6.0.x to 6.5 is not supported. You have to upgrade to 6.1.x before you upgrade to 6.5 (in my case that’s not required since already running on 6.1.1.x)

·         Any vCenter upgrade will break SRM until both sides are on the same level. I have Array replication active in case of a disaster while upgrading SRM.

Prerequisites –

·         Backup of vCenter Database on primary and recovery site.

·         Backup of SRM vPostgres Database on primary and recovery sit. (Explained in detail below)

·         Primary and Recovery Site Platform Services Controller and vCenter server instances must be running.

Order of Upgrade –

·         Upgrade Avamar to 7.5

·         RPA 4.4.1 is not compatible with ESXi 6.5 hence upgrade that to RPA 5.1

·         Upgrade vCenter Server to VCSA 6.5 GA at primary site.

·         Upgrade SRM to 6.5 at primary site. Note: SRM cannot be upgraded from 6.1.1 to 6.5.1

·         Upgrade SRA at primary site – Not required since running on latest

·         vCenter Server to VCSA 6.5 GA at recovery site.

·         Upgrade SRM to 6.5 at recovery site.
·         Upgrade SRA at recovery site – Not required since running on latest.

·         Upgrade vCenter from 6.5 GA to 6.5U1g at primary site.

·         Upgrade SRM from 6.5 to 6.5.1 at primary site.

·         Upgrade vCenter Server to VCSA 6.5U1g at recovery site.
·         Upgrade SRM from 6.5 to 6.5.1 at recovery site.
·         Verify connection between SRM. Verify Protection groups and recovery plans are valid.
·         Upgrade ESXi to 6.5, 7388607 at recovery site.

·         Upgrade ESXi to 6.5, 7388607 at primary site.

·         Upgrade virtual hardware and then VMtools on Virtual Machines – Can be scheduled during the next available outage window.

·         Upgrade XIOS and XMS to 6.0.1

Backup & Restore (if required) the SRM Embedded vPostgres Database -

1)      Log into the system on which you installed Site Recovery Manager Server.

2)      Stop the Site Recovery Manager service.

3)      Navigate to the folder that contains the vPostgres commands.

4)      If you installed Site Recovery Manager Server in the default location, you find the vPostgres commands in C:\Program Files\VMware\VMware vCenter Site Recovery Manager Embedded Database\bin.

5)      Create a backup of the embedded vPostgres database by using the pg_dump command.

pg_dump -Fc --host 127.0.0.1 --port port_number --username=db_username srm_db > srm_backup_name. To create a backup you need the admin password. We did not have the Admin password documented. Here is a link on how to reset the Admin password - http://virtuallycurious.blogspot.com/2018/06/the-case-of-forgotten-site-recovery.html

You set the port number, username, and password for the embedded vPostgres database when you installed Site Recovery Manager. The default port number is 5678. The database name is srm_db and cannot be changed.

6)      Start the Site Recovery Manager service.

7)      Restore (if things go south) by using the pg_restore command

pg_restore -Fc --host 127.0.0.1 --port port_number --username=db_username --dbname=srm_db srm_backup_name

References –

·         VMware Product Interoperability Matrices -http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php

·         Update sequence for vSphere 6.5 and its compatible VMware products (2147289) - https://kb.vmware.com/s/article/2147289

·         Backup and Restore the embedded vPostgres Database - https://docs.vmware.com/en/Site-Recovery-Manager/6.5/srm-install-config-6-5.pdf

·        EMC Recoverpoint SRA compatibility Matrix - https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=sra&productid=39129

·        Compatibility Matrix for SRM 6.5 - https://www.vmware.com/support/srm/srm-compat-matrix-6-5.html

·        Order of upgrading vSphere and SRM components - https://docs.vmware.com/en/Site-Recovery-Manager/6.5/com.vmware.srm.install_config.doc/GUID-E7B47738-C63D-4A05-9A13-7C5FF20801A7.html

·        Avamar Comaptiblity Matrix - https://support.emc.com/docu32263_Avamar-Compatibility-and-Interoperability-Matrix.pdf?language=en_US