Virtually Curious: March 2018

Thursday, March 29, 2018

How to find the disk size for an Avamar BMR restore

We had a scenario where a BMR (Bare Metal Restore) system state restore was required.

A skeleton VM was created with drives matching the original server and booted from the Avamar BMR ISO. The restore was kicked off and after 90% the restore failed. Why?

Because the disk sizes did not match. The VM had several disks, some were stripped volumes some as spanned volumes. When the restore skeleton VM was created, the Admins just matched the volumes to the old VM. Avamar has no visiblity of how the disks are laid out within the OS.

Example of how the disks were laid out in VMware and inside the OS -

So the question is how do you find the disk size if the VM is down and you need a BMR restore?

Avamar has a very nice Avtar command line utility. I could not find an official document from EMC but there is bits and pieces of information on the web.

avtar.exe --help will give you a wide range of options.

Steps to find the drive size for a BMR restore -

On your Windows workstation, install the Avamar client software and do not register it with the Avamar console.
The default installation path will be - C:\Program Files\avs\
Run the following command

C:\Program Files\avs\bin>avtar.exe -x --server=IP_of_Avamar_Server --id=Username --ap=
Password --path=/clients/servername_FQDN

--labelnum=label_number_of_the_backup_to_be_restored --internal --target=.\tmp\ .system_info

--target=.\tmp will create a tmp directory under avs\bin and the output of the command will be under this directory.

There will be numerous XML files under \tmp.
The useful files are CriticalVolumesMapping.xml and partitiontables.xml

CriticalVolumesMapping.xml will give you the details of how the disks are laid out within the OS.

Example -
-<VolumeMappings Version="2.0">

We can clearly see that -

Disks 3,4,5,6 make up Logical volume E:

Disks 8,9 make up Logical volume I:

partitiontables.xml will provide you with the disk sizes

Example -

-<PhysicalDisk NumPartitions="4" DiskSize_bytes="274872407040" PartioningScheme="MBR" MBRSignature="1720029347" DiskType="Fixed" DiskNumber="8" DiskSize_Gbytes="255" SectorSize_bytes="512">

-<PartitionList>

<Partition Size_bytes="274876826112" Start_bytes="32256" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="0" Size_Gbytes="255" Type="Alternate Linux swap" Bootable="false"/>

<Partition Size_bytes="0" Start_bytes="0" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="1" Size_Gbytes="0" Type="Empty" Bootable="false"/>

<Partition Size_bytes="0" Start_bytes="0" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="2" Size_Gbytes="0" Type="Empty" Bootable="false"/>

<Partition Size_bytes="0" Start_bytes="0" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="3" Size_Gbytes="0" Type="Empty" Bootable="false"/>

</PartitionList>

</PhysicalDisk>

-<PhysicalDisk NumPartitions="4" DiskSize_bytes="274872407040" PartioningScheme="MBR" MBRSignature="1720029346" DiskType="Fixed" DiskNumber="9" DiskSize_Gbytes="255" SectorSize_bytes="512">

-<PartitionList>

<Partition Size_bytes="0" Start_bytes="0" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="1" Size_Gbytes="0" Type="Empty" Bootable="false"/>

<Partition Size_bytes="0" Start_bytes="0" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="2" Size_Gbytes="0" Type="Empty" Bootable="false"/>

<Partition Size_bytes="0" Start_bytes="0" partStyle="MBR" SerialNumber_dec="0" SerialNumber_hex="0" PartitionNumber="3" Size_Gbytes="0" Type="Empty" Bootable="false"/>

In the above scenario, Disks 8 and 9 together make up Volume I: and from the partitiontables.xml file you know the total size of I: will be 510GB. While creating the skeleton VM you will create a 510 GB drive.

Once you create the exact size and number of drives required, Avamar will perform a restore without any hiccups.

Problem solved !

Thursday, March 15, 2018

VMware Stack Upgrade

I’m working on a plan to upgrade our existing VMware Stack and wanted to write a detailed post about all components involved and the order of upgrade.

Current State –

· Primary and Recovery site with XtremIO Arrays on both ends with physical RecoverPoint appliances for array based replication.

· vCenter Servers on both sites (with embedded Platform Services Controller) running on Windows Server 2012 R2 with an external SQL Database.

· ESXi 6.0.0, 3825889

· Site Recovery Manager (SRM) 6.1.1.13825

· Storage Replication Adapter (SRA) 2.2.0.3

· XtremIO (XIOS) 4.0.15-24, XMS 4.2.1

· RecoverPoint (RPA) 4.4.1

· Avamar 7.3

Future State –

· vCenter Server Appliance (VCSA) 6.5 U1f

· ESXi 6.5, 7388607

· SRM 6.5.1, 6014840

· SRA 2.2.0.3

· XIOS 6.0.1, XMS 6.0.1

· RecoverPoint (RPA) 5.1

· Avamar 7.5

There are numerous inter-dependencies to get to the future state.

· Avamar 7.3 is not compatible with vCenter 6.5

· RPA 4.4.1 is not compatible with ESXi 6.5

· Upgrading SRM from 6.0.x to 6.5 is not supported. You have to upgrade to 6.1.x before you upgrade to 6.5 (in my case that’s not required since already running on 6.1.1.x)

· Any vCenter upgrade will break SRM until both sides are on the same level. I have Array replication active in case of a disaster while upgrading SRM.

Prerequisites –

· Backup of vCenter Database on primary and recovery site.

· Backup of SRM vPostgres Database on primary and recovery sit. (Explained in detail below)

· Primary and Recovery Site Platform Services Controller and vCenter server instances must be running.

Order of Upgrade –

· Upgrade Avamar to 7.5

· RPA 4.4.1 is not compatible with ESXi 6.5 hence upgrade that to RPA 5.1

· Upgrade vCenter Server to VCSA 6.5 GA at primary site.

· Upgrade SRM to 6.5 at primary site. Note: SRM cannot be upgraded from 6.1.1 to 6.5.1

· Upgrade SRA at primary site – Not required since running on latest

· vCenter Server to VCSA 6.5 GA at recovery site.

· Upgrade SRM to 6.5 at recovery site.
· Upgrade SRA at recovery site – Not required since running on latest.

· Upgrade vCenter from 6.5 GA to 6.5U1g at primary site.

· Upgrade SRM from 6.5 to 6.5.1 at primary site.

·         Upgrade vCenter Server to VCSA 6.5U1g at recovery site.
·         Upgrade SRM from 6.5 to 6.5.1 at recovery site.
·         Verify connection between SRM. Verify Protection groups and recovery plans are valid.
· Upgrade ESXi to 6.5, 7388607 at recovery site.

· Upgrade ESXi to 6.5, 7388607 at primary site.

· Upgrade virtual hardware and then VMtools on Virtual Machines – Can be scheduled during the next available outage window.

· Upgrade XIOS and XMS to 6.0.1

Backup & Restore (if required) the SRM Embedded vPostgres Database -

1) Log into the system on which you installed Site Recovery Manager Server.

2) Stop the Site Recovery Manager service.

3) Navigate to the folder that contains the vPostgres commands.

4) If you installed Site Recovery Manager Server in the default location, you find the vPostgres commands in C:\Program Files\VMware\VMware vCenter Site Recovery Manager Embedded Database\bin.

5) Create a backup of the embedded vPostgres database by using the pg_dump command.

pg_dump -Fc --host 127.0.0.1 --port port_number --username=db_username srm_db > srm_backup_name. To create a backup you need the admin password. We did not have the Admin password documented. Here is a link on how to reset the Admin password - http://virtuallycurious.blogspot.com/2018/06/the-case-of-forgotten-site-recovery.html

You set the port number, username, and password for the embedded vPostgres database when you installed Site Recovery Manager. The default port number is 5678. The database name is srm_db and cannot be changed.

6) Start the Site Recovery Manager service.

7) Restore (if things go south) by using the pg_restore command

pg_restore -Fc --host 127.0.0.1 --port port_number --username=db_username --dbname=srm_db srm_backup_name

References –

· VMware Product Interoperability Matrices -http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php

· Update sequence for vSphere 6.5 and its compatible VMware products (2147289) - https://kb.vmware.com/s/article/2147289

· Backup and Restore the embedded vPostgres Database - https://docs.vmware.com/en/Site-Recovery-Manager/6.5/srm-install-config-6-5.pdf

· EMC Recoverpoint SRA compatibility Matrix - https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=sra&productid=39129

· Compatibility Matrix for SRM 6.5 - https://www.vmware.com/support/srm/srm-compat-matrix-6-5.html

· Order of upgrading vSphere and SRM components - https://docs.vmware.com/en/Site-Recovery-Manager/6.5/com.vmware.srm.install_config.doc/GUID-E7B47738-C63D-4A05-9A13-7C5FF20801A7.html

· Avamar Comaptiblity Matrix - https://support.emc.com/docu32263_Avamar-Compatibility-and-Interoperability-Matrix.pdf?language=en_US