Ruby vSphere Console (RVC) vSAN Admins Friend

Ruby vSphere Console (RVC) is console interface, it help to manage, monitor vSAN environment.

RVC comes bundled with vCenter and no additional installation is required.

Its is available for Windows based vCenter and vCenter Appliance too.

RVC will help you manage your VSAN servers, it will provide you detailed view of VSAN servers, you can also run VSAN observer service using RVC.

How to login to RVC?

1. Enable SSH on vCenter Appliance server.

2. Login to vCenter using SSH

3. Enter shell command to go to bash shell on vCenter
     >shell

4. On bash prompt Enter below command to login to RVC console and hit Enter and then Enter SSO Administrator user password to login.
# rvc Administrator@vsphere.local@localhost

e.g. 
root@dc1-vcenter01 [ ~ ]# rvc Administrator@vsphere.local@localhost
Install the "ffi" gem for better tab completion.
password:
0 /
1 localhost/
>

Navigating using RVC

#cd command to change directory, select the object number to cd it.
e.g. cd 1
#ls command to display items and then use cd command.
#mark - using mark command you create alias of any object.
e.g. in below command am creating ~cluster alias for my vsan cluster.

#mark cluster /localhost/AllenDC-VA/computers/vsan01

to use alias, you need to add ~ in front of alias/mark
#vsan.check_state ~cluster

root@dc1-vcenter01 [ ~ ]# rvc Administrator@vsphere.local@localhost
Install the "ffi" gem for better tab completion.
password:
0 /
1 localhost/
> cd 1
/localhost> ls
0 ChiDC-IL (datacenter)
1 MalDC-PA (datacenter)
2 VanDC, BC (datacenter)
3 Decomission (datacenter)
4 DCtown-MO (datacenter)
5 DCeigh-NC (datacenter)
6 DCton-MA (datacenter)
7 AllenDC-VA (datacenter)
/localhost> cd 7
/localhost/AllenDC-VA> ls
0 storage/
1 computers [host]/
2 networks [network]/
3 datastores [datastore]/
4 vms [vm]/
/localhost/AllenDC-VA> cd 1
/localhost/AllenDC-VA/computers> ls
0 gla-vsan01 (cluster): cpu 40 GHz, memory 215 GB
/localhost/AllenDC-VA/computers> cd 0
/localhost/AllenDC-VA/computers/gla-vsan01> ls
0 hosts/
1 resourcePool [Resources]: cpu 40.61/40.61/normal, mem 215.54/215.54/normal
/localhost/AllenDC-VA/computers/gla-vsan01> mark cluster /localhost/AllenDC-VA/computers/gla-vsan01
/localhost/AllenDC-VA/computers/gla-vsan01> vsan.check_state ~cluster

or you can also below command. ( notice . after command ) 
/localhost/AllenDC-VA/computers/gla-vsan01> vsan.check_state .


Useful VSAN Commands

vsan.disks_stats - management of disk groups and monitoring the health of physical disks
vsan.check_state - troubleshooting data unavailability situations and understanding object health in the VSAN cluster
vsan.resync_dashboard - Check data resync progress, visibility into data resync when changing storage policies
vsan.whatif_host_failures - understanding VSAN’s ability to tolerate node failures
vsan.proactive_rebalance - start proactive data re-balance on physical disks.
vsan.proactive_rebalance_info - check status of proactive data re-balance.

vsan.check_limits - check vsan object limits
vsan.cluster_info  - get detailed information about vsan cluster
vsan.disk_object_info - get information about all vSAN objects on a given physical disk
vsan.disks_info - get physical disk information
vsan.vm_object_info - get vsan information of a VM
vsan.vmdk_stats - get vmdkdetails.
vsan.observer ~cluster --run-webserver --force  - start web service, it collect VSAN performance statistics data. you can access vsan observer performance data by vising to https://vCenterServer_hostname_or_IP_Address:8010 URL.

How to Upgrade vSAN ESXi Hosts

ESXi hosts participating in VSAN Cluster can be upgrading using Update manager or using ESXCLI as you would do any other ESXi host upgrade. However, VSAN Cluster need special attention before you plan upgrade.

You need to be more careful while upgrading ESXi Hosts and make sure you have supported Hardware Raid Controller, you install supported firmware/drivers, supported disk firmware…etc.

If you are not sure about anything do not procced with upgrade and Create a Support request with Vmware Support Team.

You can perform vSAN health check using vSAN health plugin and RVC console.

With new version of vSphere, Vmware has integrated update manager with vSAN. Based on your vSAN configuration, update manager creates patch baseline and assign it to vSAN cluster.

Do the upgrade precheck before upgrading each vSAN ESXi host. You can do the precheck using web client and also using RVC.

I have divided upgrade process in four parts. see the step by step process to do the vSAN cluster ESXi hosts upgrade.

How to Upgrade vSAN ESXi Hosts

• Part 1 - vSAN ESXI host Upgrade Precheck Tasks
• Part 2 - vSAN ESXi Host Upgrade Using Update Manager
• Part 3 - vSAN ESXi Host upgrade issues
• Part 4 - vSAN On-Disk Format Upgrade

How to Get ESXi Boot Device

How to get ESXi boot device?

If you need to see the current boot device of ESXi or from which device ESXi is running then follow below steps.

1. Enable SSH on ESXi host.

2. Login to ESXi server using root and its password.

3. Run below Command to get boot volume

#ls -l /bootbank | awk -F"-> " '{print $2}'

E.g output

[root@esxi01:~] ls -l /bootbank | awk -F"-> " '{print $2}'

/vmfs/volumes/50705ae3-b08ce25f-a1b8-bc56258253ad

4. Copy vmfs volume and run below command to get more details about volume.

#vmkfstools -P vmfsVolumeID

[root@esxi01:~] vmkfstools -P /vmfs/volumes/50705ae3-b08ce25f-a1b8-bc56258253ad

vfat-0.04 (Raw Major Version: 0) file system spanning 1 partitions.

File system label (if any):

Mode: private

Capacity 261853184 (63929 file blocks * 4096), 90677248 (22138 blocks) avail, max supported file size 0

UUID: 50705ae3-b08ce25f-a1b8-bc56258253ad

Partitions spanned (on "disks"):

       eui.00a0504658335330:5

Is Native Snapshot Capable: NO

5. From above command output copy the disks/partition UID and run below command to get device details.

#esxcli storage core device list | grep -A27 deviceID

[root@esxi01:~] esxcli storage core device list | grep -A27 eui.00a0504658335330

eui.00a0504658335330

  Display Name: Local USB Direct-Access (eui.00a0504658335330)

  Has Settable Display Name: false

  Size: 30436

  Device Type: Direct-Access

  Multipath Plugin: NMP

  Devfs Path: /vmfs/devices/disks/eui.00a0504658335330

  Vendor: Cypress

  Model: SDRAID

  Revision: 0000

  SCSI Level: 2

  Is Pseudo: false

  Status: on

  Is RDM Capable: false

  Is Local: true

  Is Removable: true

  Is SSD: false

  Is VVOL PE: false

  Is Offline: false

  Is Perennially Reserved: false

  Queue Full Sample Size: 0

  Queue Full Threshold: 0

  Thin Provisioning Status: unknown

  Attached Filters:

  VAAI Status: unsupported

  Other UIDs: vml.0000000000766d68626133323a303a30

  Is Shared Clusterwide: false

  Is Local SAS Device: false

  Is SAS: false

  Is USB: true

  Is Boot USB Device: true

  Is Boot Device: true

  Device Max Queue Depth: 1

  No of outstanding IOs with competing worlds: 32

In above command you can see, this ESXi host is booted and running from USB device.

Above process looks simple and useful.

If you have large number of ESXi hosts and want to export all ESXi hosts boot device then you can automate it using PowerShell and bash scripting.

How to Export ESXi Boot Device?

1. Enable SSH on all ESXi hosts for which you want to get boot device. You can use PowerShell script to automate enabling SSH service on many esxi hosts at once.

2. Create a text file with below command and save it as c:\temp\ESXi-boot-device-cmd.txt

b=`ls -l /bootbank | awk -F"-> " '{print $2}'`

d=`vmkfstools -P $b | egrep 'mpx|naa|eui' | awk -F":" '{print $1 }' | sed -e 's/^[ \t]*//'`

esxcli storage core device list | grep -A27  $d | egrep 'Display Name:|Size:|Multipath Plugin:|Devfs Path:|Vendor:|Model:|Revision:|Is RDM Capable:|Is Local:|Is Removable:|Is Shared Clusterwide:|Is USB:|Is Boot USB Device:|Is Boot Device:' | awk -F":" '{print $2}' |sed  's/ //g' | awk -vRS="" -vOFS=',' '$1=$1'

3. Download plink.exe and save it in c:\temp\ folder

https://the.earth.li/~sgtatham/putty/latest/w32/plink.exe

Plink is a command line connection tool similar to unix SSH.

What is plink?

http://the.earth.li/~sgtatham/putty/0.53b/htmldoc/Chapter7.html

4. Below is a PowerShell script to Export boot device of all ESXi hosts.

Change below variable in script -

$folder - folder where you have saved esxi command file, plink.exe

$vcenter - change vcenter name

$password - enter your ESXi password.

This script connect to all ESXi hosts which are connected in vCenter, then it connect to each ESXi hosts using SSH and execute ESXi command and finally it exports boot device to $vcenter-boot-device-info.csv file.

$folder = "C:\temp\"

$vcenters = "vcenter01"

$password = 'EsxiPassword'

foreach ( $vcenter in $vcenters ) {

   connect-viserver $vcenter

   $vmhosts = get-vmhost | where { $_.connectionState -ne "NotResponding" } | sort Name

$report = @()

foreach ( $vmhost in $vmhosts.name ) {

   $details = " " | select HostName,DeviceName,SettableDisplay,Size,MPP, DevfSPath, Vendor,Model,Revision,RDM,Local,Removable,QueueSize,isShared,isUSB,BootUSB,isBootDevice

   $details.Hostname = $vmhost

   echo y | $folder\plink.exe  -ssh root@$vmhost -pw $password hostname

   $hostdetails = $folder\plink.exe  -ssh root@$vmhost -pw $password -m $folder\cmd.txt

   $details.devicename = $hostdetails

   $report += $details
   }
   $report | export-csv -NoTypeInformation -path $folder\$vcenter-boot-device-info.csv
}

Once you get CSV, you would need to open it in Excel and select all cell from B2 and do text to column and separate all cells from B2 text with comma as delimiter.   

E.g. output