Thursday 22 February 2018

vSAN Encryption Implementation & Monitoring

vSAN Encryption Implementation & Monitoring
Contents
vSAN Encryption – useful documents/posts   
KMS Server   
vSAN Encryption Monitoring and Reporting   
How to monitor vSAN Encryption using vCenter?   
How to Monitor vSAN encryption using PowerCLI?   
vSAN Encryption Report using PowerCLI Script   
How to find Which KMS Key Encryption Key (KEK) vSAN Cluster is Using? 

vSAN Encryption – useful documents/posts

 
I found below posts, very useful while implementing vSAN Encryption. So, I would suggest you refer it if you are looking to implement and manage vSAN Encryption or VM Encryption.

KMS Server

To enable vSAN Encryption we need third party Key Management server. Am using HyTrust key Control server and I found it very easy to deploy, manage, upgrade.
Deploy 2 nodes of KMS server in one KMS cluster to have the high availability.  

vSAN Encryption Monitoring and Reporting

 
Once you have enabled vSAN Encryption for vSAN Cluster then you monitor vSAN encryption using below steps.

How to monitor vSAN Encryption using vCenter?

  1. Check vCenter and KMS server’s connection Status.
If connection statue is not Normal, make sure vCenter is connected to KMS and there are no trust related issues.
 
From vCenter web client select vCenter in left pane Select Configure tab Select Key Management Server

  1. Select ESXi Cluster where you have enabled vSAN Encryption. vSAN Health plugin see the KMS connectivity status and Disk Encryption status.
If you have enabled Encryption then Encryption links are displayed in vSAN Health plugin.
 
  1. CPU AES-NI feature status

  1. Check KMS and vCenter status

  1. Check KMS and ESXi Host Connection Status

You can see vCenter is showing little information about vSAN Encryption.  So, we need something on command line.

How to Monitor vSAN encryption using PowerCLI?

 
I would highly recommend watching below VMworld session if you are looking to Implement VM Encryption or vSAN Encryption.

vSAN Encryption Report using PowerCLI Script

 
Jase McCarty has created a very good PowerCLI script Vsan-EncryptionReport.ps1 which provide detailed status of vSAN Encryption.  You can schedule this script to send weekly health check report of vSAN Encryption enabled cluster or run it periodically as needed.
You can download many of his useful scripts from github.
You require PowerCLI 6.5.1 and vSAN 6.6 with Encryption enabled to use this script.
Make sure you have required PowerCLI version, with older PowerCLI this script will not work.
Download Vsan-EncryptionReport.ps1 and save it on your local machine.
Start PowerCLI/PowerShell with VMware PowerCLI Modules.
Go to the folder where script is stored and execute script. ( see known issues below )
PS > .\Vsan-EncryptionReport.ps1 -vCenter vlab-vcenter -ClusterName "vlab-vsan03"

This script will connect to vCenter and get the Encryption information for specified cluster.
e.g.  vSAN Encryption Report for my test cluster.  
Name                           Port  User                          
----                           ----  ----                          
vlab-venter               443   domain\vcenteruser         
vlab-vsan03 is Encrypted
vSAN Encryption Results
*******************************************************************
Overall/Config
Overall Health:                         red
Config Health:                          red
*******************************************************************
KMS
KMS Health:                             yellow
*******************************************************************
vCenter Results
vCenter KMS Provider:                        vlabKMS01
vCenter KMS Health:                          yellow
vCenter KMS Client Certificate Health:       green
vCenter KMS Client Certificate Expire Date:  29-Jan-19 12:15:08 PM
*******************************************************************
Host Results
    vlab-esxi1.domain.com
    Overall KMS Health:             yellow
                 Issue:             servercertificatesinconsistent
    AES-NI Enabled:                 True
    *******************************************************************
     5  Total Disks
     5  Disks without Encryption Issues
    DekGenerationID  Name
        2                mpx.vmhba1:C0:T2:L0
        2                mpx.vmhba1:C0:T3:L0
        2                mpx.vmhba1:C0:T4:L0
        2                mpx.vmhba1:C0:T0:L0
        2                mpx.vmhba1:C0:T1:L0
    *******************************************************************
    vlab-esxi2.domain.com
    Overall KMS Health:             yellow
                 Issue:             servercertificatesinconsistent
    AES-NI Enabled:                 True
    *******************************************************************
     5  Total Disks
     5  Disks without Encryption Issues
    DekGenerationID  Name
        2                mpx.vmhba1:C0:T1:L0
        2                mpx.vmhba1:C0:T2:L0
        2                mpx.vmhba1:C0:T0:L0
        2                mpx.vmhba1:C0:T4:L0
        2                mpx.vmhba1:C0:T3:L0
    *******************************************************************
    vlab-esxi3.domain.com
    Overall KMS Health:             yellow
                 Issue:             servercertificatesinconsistent
    AES-NI Enabled:                 True
    *******************************************************************
     5  Total Disks
     5  Disks without Encryption Issues
    DekGenerationID  Name
        2                mpx.vmhba1:C0:T0:L0
        2                mpx.vmhba1:C0:T2:L0
        2                mpx.vmhba1:C0:T4:L0
        2                mpx.vmhba1:C0:T1:L0
        2                mpx.vmhba1:C0:T3:L0
    *******************************************************************
    *******************************************************************
    Summary
    *******************************************************************
    Total Hosts with issues:                   3
    Total Hosts without issues:                
    Total Disks with issues:                   0
    Total Disks without issues:                15

In above report you can see vSAN Host disks are encrypted and are working fine. However, there is some issue with KMS connection.
If Everything is working fine then this report would show everything green.
 
Known issues
  1. While executing the script for the first time, I had issues with PowerCLI. After some attempts I could install latest required PowerCLI.
  1. Script was not displaying any output – to resolve this issue by modifying the condition as below  
# If vSAN is enabled and it is Encrypted
#If($Cluster.vSanEnabled -And $EncryptedVsan.EncryptionEnabled){   comment it
If($Cluster.vSanEnabled ){                         new Line
 
 How to find Which KMS Key Encryption Key (KEK) vSAN Cluster is Using?
 
After implementing vSAN Encryption I was just curious to know which Key Encryption Key is being used by my vsan cluster. First, we need to understand we cannot see the actual KEK anywhere but we can see the KEK UUID in HyTrust GUI and in ESXi Host cache.
After some googling, I found VMworld 2017 session of James Doyle in which he has demonstrated how to find KEK UUIDs for VM Encryption. (Link is above, must see)
Make sure you have latest PowerCLI which has VM Encryption Module.  For more information visit - PowerCLI for VM Encryption - https://blogs.vmware.com/vsphere/2016/12/powercli-for-vm-encryption.html
I used Crypto_Key_Summary.ps1 to find KEK UUIDs for vSAN Encryption and able to get the KEK UUIDs.
Download the script file from above Github and save it on local machine.
You would need to make some modification to script and add your vCenter Name, username, password …etc.  Once ready, execute the script to get the encryption details from vCenter.
e.g. output for vSAN Cluster. I have not enabled VM encryption so there is nothing under VM encryption.
Name                           Port  User                          
----                           ----  ----                          
vlab-vcenter                   443   domain\vcenteruser         
*********
*vCenter*
*********
Key IDs vCenter is aware of:
 
 
KeyID                                KMS_ClusterId
-----                                -------------
8e339447-04f1-11e8-xxxx-xxxxxxx vlabKMS01    
  
 
************************
*Host Key Cache Summary*
************************
 
Hostname: vlab-esxi1.domain.com
KeyIds in host's cache
8e339447-04f1-11e8-xxxx-xxxxxxxx
KMS Cluster providing keys
vlabKMS01
 
Hostname: vlab-esxi2.domain.com
KeyIds in host's cache
8e339447-04f1-11e8-xxxx-xxxxxxxx
KMS Cluster providing keys
vlabKMS01
 
Hostname: vlab-esxi3.domain.com
KeyIds in host's cache
8e339447-04f1-11e8-xxxx-xxxxxxxx
KMS Cluster providing keys
vlabKMS01
 
************************************
*HostKeys used by Crypto-safe Hosts*
************************************
vlab-esxi1.domain.com
 
KeyID                                KMS_ClusterId
-----                                -------------
8e339447-04f1-11e8-xxxx-xxxxxxxx    vlabKMS01    
 
 
vlab-esxi2.domain.com
 
KeyID                                KMS_ClusterId
-----                                -------------
8e339447-04f1-11e8-xxxx-xxxxxxxx    vlabKMS01    
 
 
vlab-esxi3.domain.com
 
KeyID                                KMS_ClusterId
-----                                -------------
8e339447-04f1-11e8-xxxx-xxxxxxxx    vlabKMS01    
 
 
 
****************************
*Keys used by Encrypted VMs*
****************************

In above output, we can see KeyID, now you can take this Key ID and search it in KMS server GUI, in my case HyTrust GUI.



Thanks,
vPRH