Replacing SSL Certificates VMware vCenter 6.0 Update 2

SSL certificates are the one of the most important part in VMware vCenter server and VMware ESXi. All traffic between ESXi Host, vCenter and between all vCenter services are encrypted using SSL certificates. Over the time VMware has improved the process to replace SSL certificates for different vCenter components.

If you need to update SSL certificates of vCenter 6.0 then you can follow below KB to replace certificates.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2111219

However the process to replace SSL certificate of vCenter 6.0 Update 1b is changed and is not well documented, so let’s See How to Replace SSL Certificates of VMware vCenter 6.0 Update 2.

In VMware vCenter 6 VMware has introduced VMware Certificate Authority (VMCA), We are going to make this VMCA as an intermediate CA of our Internal Corporate Microsoft CA server. Then VMCA will take care of replacing SSL certificates of different vCenter services and also it will issue certificates to ESXi Hosts which you add in vCenter.

Important – In vSphere SSL configuration OrgUnit need to be unique for each service or certificate replacement will fail.

In this process, we will be using below Organization Unit for vCenter services.

MACHINE_SSL_CERT.cfg  :  Root
machine.cfg           :  Machine
vsphere-webclient.cfg :  WebClient
vpxd.cfg              :  VPXD
vpxd-extension.cfg    :  VPXD-EXT
certool.cfg           :  IT-VMCA

See this Post for common issues during certificate replacement.

Error while performing Cert Replacement operation for vCenter 6

 

How to make VMCA as Intermediate CA?

1. Login to VMware vCenter Appliance using SSH with root username and password. You can use Putty utility to login to Linux system using SSH.

You need to have ssh enabled on vCenter Appliance.

Follow this document to enable SSH on vCenter server, if it is not enabled already.

http://blog.ukotic.net/2015/06/14/enable-ssh-on-vcenter-server-appliance-6-vcsa/

2. Once you logged in to vCSA using SSH, enable shell access with below commands

>shell.set --enabled True

>shell

3. Edit cartool.cfg file

#cd /usr/lib/vmware-vmca/share/config

backup cartool.cfg file before editing, so that you can revert back to original if needed.

#cp certool.cfg certool.cfg.bak

#vi certool.cfg

Press i to enter in edit mode.

Move cursor to specific line and edit details as below, Press Esc key and then :wq to save and close the file.

#
# Template file for a CSR request
#

# Country is needed and has to be 2 characters
Country = US
Name    = vlab-vc01.vprhlabs.com
Organization = your-ognization-name
OrgUnit = IT
State = Missouri
Locality = Raytown
IPAddress = 192.168.10.10   
Email = vprh@vprhlabs.com
Hostname = vlab-vc02.vprhlabs.com

4. Execute certificate-manager command to start SSL Certificate replacement process.

#/usr/lib/vmware-vmca/bin/certificate-manager

5. Select option 2 to Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates and Press Enter.

Follow the prompt and provide your inputs, I have highlighted my inputs. Also to accept the default values [values in name bracket] press Enter

6. Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y

7. Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:****      you cannot see password * characters on Linux, enter correct password.

8. Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

• Enter proper value for 'Country' [Default value : US] :
• Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
• Enter proper value for 'Organization' [Default value : vPRHLABS] :
• Enter proper value for 'OrgUnit' [Default value : ITS] : Root
• Enter proper value for 'State' [Default value : North Carolina] :
• Enter proper value for 'Locality' [Default value : Raleigh] :
• Enter proper value for 'IPAddress' [optional] : 192.168.105.10
• Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
• Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com
  

Continue the the wizard to create machine.cfg file.

9. Please configure machine.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

• Enter proper value for 'Country' [Default value : US] :
• Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
• Enter proper value for 'Organization' [Default value : vPRHLABS] :
• Enter proper value for 'OrgUnit' [Default value : ITS] : Machine
• Enter proper value for 'State' [Default value : North Carolina] :
• Enter proper value for 'Locality' [Default value : Raleigh] :
• Enter proper value for 'IPAddress' [optional] : 192.168.105.10
• Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
• Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com
  

Continue the the wizard, to create vsphere-webclient.cfg file.

10. Please configure vsphere-webclient.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

• Enter proper value for 'Country' [Default value : US] :
• Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
• Enter proper value for 'Organization' [Default value : vPRHLABS] :
• Enter proper value for 'OrgUnit' [Default value : ITS] : WebClient
• Enter proper value for 'State' [Default value : North Carolina] :
• Enter proper value for 'Locality' [Default value : Raleigh] :
• Enter proper value for 'IPAddress' [optional] : 192.168.105.10
• Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
• Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.corp.allscripts.com
  

Please configure vpxd.cfg with proper values before proceeding to next step.

11. Press Enter key to skip optional parameters or use Default value.

• Enter proper value for 'Country' [Default value : US] :
• Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
• Enter proper value for 'Organization' [Default value : vPRHLABS] :
• Enter proper value for 'OrgUnit' [Default value : ITS] : VPXD
• Enter proper value for 'State' [Default value : North Carolina] :
• Enter proper value for 'Locality' [Default value : Raleigh] :
• Enter proper value for 'IPAddress' [optional] : 192.168.105.10
• Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
• Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

Continue the wizard to create vpxd-extension.cfg

12. Please configure vpxd-extension.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

• Enter proper value for 'Country' [Default value : US] :
• Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
• Enter proper value for 'Organization' [Default value : vPRHLABS] :
• Enter proper value for 'OrgUnit' [Default value : ITS] :VPXD-EXT
• Enter proper value for 'State' [Default value : North Carolina] :
• Enter proper value for 'Locality' [Default value : Raleigh] :
• Enter proper value for 'IPAddress' [optional] : 192.168.105.10
• Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
• Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

13. Continue the wizard and select option 1 to generate Certificate Signing request and key for VMCA root certificate.

• 1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
• 2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate

Option [1 or 2]: 1

14. Please provide a directory location to write the CSR(s) and PrivateKey(s) to:
Output directory path: /root/

15. Please configure certool.cfg with proper values before proceeding to next step.

• Press Enter key to skip optional parameters or use Default value.
• Enter proper value for 'Country' [Default value : US] :
• Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] : VMCA
• Enter proper value for 'Organization' [Default value : vPRHLABS] :
• Enter proper value for 'OrgUnit' [Default value : ITS] : IT-VMCA
• Enter proper value for 'State' [Default value : North Carolina] :
• Enter proper value for 'Locality' [Default value : Raleigh] :
• Enter proper value for 'IPAddress' [optional] : 192.168.105.10
• Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
• Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

16. This will generate Certificate Signing request (CSR) and key for VMCA certificate and store at the location specified e.g. /root/ as specified above.

2016-05-30T16:51:51.299Z   Running command: ['/usr/lib/vmware-vmca/bin/certool', '--genkey', '--privkey', '/root/vmca_issued_key.key', '--pubkey', '/tmp/pubkey.pub']
2016-05-30T16:51:51.937Z   Done running command
2016-05-30T16:51:51.937Z   Running command: ['/usr/lib/vmware-vmca/bin/certool', '--gencsr', '--privkey', '/root/vmca_issued_key.key', '--pubkey', '/tmp/pubkey.pub', '--config', '/var/tmp/vmware/certool.cfg', '--csrfile', '/root/vmca_issued_csr.csr']
2016-05-30T16:51:52.343Z   Done running command

CSR generated at: /root/vmca_issued_csr.csr

17. Keep the putty session running, don’t close it.

18. Login to vCenter Appliance using WinSCP and download the CSR file on your local system.

18.1. To access vCSA using WinSCP, you need to first set root shell to bash.

18.2. Open one more Putty session and login to vCenter Appliance with root user and password.

>shell.set --enabled True

>shell

#chsh -s /bin/bash root

18.3 Now Download and install WinSCP on your system.

18.4 Launch WinSCP, Enter IP or Hostname of vCenter system, Enter username and password and connect to vCenter.

18.5 Download /root/vmca_issued_csr.csr file from vCSA server to your local system.

19. Submit CSR file to your Certificate Authority to receive SSL certificate.

If you are using Microsoft CA of your organization then first you need to have certificate Template configured which can be used to issue certificates for Intermediate CA or Subordinate CA.

https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2112016

In My testing setup, am using Internal Microsoft Certificate Authority to issue certificate to VMCA. Make sure you have access to use that certificate template.

Get the CA certificate from Certificate Authority.

19.1 Open Command prompt and Change to the directory where you have downloaded CSR file.

>cd F:\tmp\vlab-vc01

>certreq -submit -config "CAServer\CAServer" -attrib "CertificateTemplate:VMCAIntermediateCA" vmca_issued_csr.csr vlab-vc01-CA.crt

• VMCAIntermediateCA  is the name of my SSL certificate Template for VMware in Microsoft CA.
• CAServer – is name of CA server, replace this value with your CA server name.

Alternatively, you can visit the CA URL and request SSL certificate.

19.2 Also you need to download complete Chain of ROOT CA certificates chain.

20. Create a Certificate chain using VMCA certificate, Intermediate CA and root CA certificate.

Open Command prompt and change to directory where you have saved all certificates.

>copy vlab-vc01-VMCA.cer+root64-2.cer+root64-1.cer root_signing_chain.cer

• root64-2.cer – is first intermediate Certificate of CA
• root64-1.cer – is root CA certificate.

You can also open certificate files using Notepad++ and copy/paste VMCA certificate, Intermediate and root CA’s certificate to create certificate chain.

Note, while copying certificate, there should not be any empty lines and you should follow below certificate sequence otherwise certificate replacement will not work.

-----BEGIN CERTIFICATE-----

Copy VMCA certificate here

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Copy Intermediate CA certificate here

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Copy Root CA certificate here

-----END CERTIFICATE-----

save this certificate chain file in root_signing_chain.cer file, make sure there are no empty lines or you don’t add any extra characters. 

21. Upload Certificate chain to vCSA server /root/ location using WinSCP.

22. Now, go back to first Putty session where we start the certificate-manager to replace certificate.

23. Press 1 to Import VMCA certificate

• 1. Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate
• 2. Exit certificate-manager

Option [1 or 2]: 1

24. Please provide valid custom certificate for Root.
File : /root/root_signing_chain.cer

Please provide valid custom key for Root.
File : /root/vmca_issued_key.key

25. You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : Y

This will import the VMCA certificate chain and then VMCA will replace SSL certificates of vCenter services.

26. Watch the certificate replacement message and make sure there are no errors.

Note, if Certificate manager failed to install new SSL certificates, it will rollback the changes and install old certs and will start services without any issues. Just wait for some time to start all services.

Get site nameCompleted [Replacing Machine SSL Cert...]
vlab-vc-site
Lookup all services
Get service vlab-vc-site:dc733106-31e4-4a48-b7ea-a0fdde9313bc
Update service vlab-vc-site:dc733106-31e4-4a48-b7ea-a0fdde9313bc; spec: /tmp/svcspec_UN174R
Get service vlab-vc-site:b91bc335-1d8d-46b8-8257-879474422483
Update service vlab-vc-site:b91bc335-1d8d-46b8-8257-879474422483; spec: /tmp/svcspec_dA3Ckh
Get service vlab-vc-site:7b9f83ed-c3da-4fd9-bed6-785fcf6a5019
Update service vlab-vc-site:7b9f83ed-c3da-4fd9-bed6-785fcf6a5019; spec: /tmp/svcspec_Wz_2kH
Get service 47c36d0a-bb88-486a-a0f5-eb7f32b8716b
Update service 47c36d0a-bb88-486a-a0f5-eb7f32b8716b; spec: /tmp/svcspec_jQc3Tm
Get service 1d1c2534-cb7e-42f9-99f2-5c08234179ee
Update service 1d1c2534-cb7e-42f9-99f2-5c08234179ee; spec: /tmp/svcspec_3CC3O_
Get service 5c7befd1-cd3b-40a1-b5da-e2ade5812fa8
Update service 5c7befd1-cd3b-40a1-b5da-e2ade5812fa8; spec: /tmp/svcspec_Gy2SCx
Get service b3c094a9-0e70-40fc-a213-b7c0f91208f4
Update service b3c094a9-0e70-40fc-a213-b7c0f91208f4; spec: /tmp/svcspec_tbdzvT
Get service aa81dd46-96f3-4d96-a055-cca9e2230e8e
Update service aa81dd46-96f3-4d96-a055-cca9e2230e8e; spec: /tmp/svcspec_IJPMbR
Get service 61bbcc34-9762-4517-8e57-f13ce25906b8
Update service 61bbcc34-9762-4517-8e57-f13ce25906b8; spec: /tmp/svcspec_uiMUqy
Get service 582aa620-b6d2-4e4f-8e3e-4d7dac838015
Update service 582aa620-b6d2-4e4f-8e3e-4d7dac838015; spec: /tmp/svcspec_mCTfhR
Get service a64b10d5-4a3f-4d5d-b370-a03e17887cc1
Update service a64b10d5-4a3f-4d5d-b370-a03e17887cc1; spec: /tmp/svcspec_bxOJ2K
Get service d3becf3f-c436-4f40-895e-8231fdbbeb38
Update service d3becf3f-c436-4f40-895e-8231fdbbeb38; spec: /tmp/svcspec_fS7RY1
Get service 61bbcc34-9762-4517-8e57-f13ce25906b8_authz
Update service 61bbcc34-9762-4517-8e57-f13ce25906b8_authz; spec: /tmp/svcspec_CbuXPJ
Get service a106c4af-1a97-41ad-9832-0ed06b31c32e
Update service a106c4af-1a97-41ad-9832-0ed06b31c32e; spec: /tmp/svcspec_YJvG5j
Get service ae8a88e1-1566-4fa7-972d-f892575dc108
Update service ae8a88e1-1566-4fa7-972d-f892575dc108; spec: /tmp/svcspec_6LAnr3
Get service 3010064d-717f-45e6-ac0d-4e2ab4063eb4
Update service 3010064d-717f-45e6-ac0d-4e2ab4063eb4; spec: /tmp/svcspec_qbll2S
Get service b3722397-a2d0-49dc-9106-2ddcc5db0e06
Update service b3722397-a2d0-49dc-9106-2ddcc5db0e06; spec: /tmp/svcspec_BgGjx1
Get service bfca6b85-0021-4ee5-84f7-275fcd4fac53
Update service bfca6b85-0021-4ee5-84f7-275fcd4fac53; spec: /tmp/svcspec_WGHmFK
Get service 3a74e01e-613c-4db8-a363-a2af7a63f073
Update service 3a74e01e-613c-4db8-a363-a2af7a63f073; spec: /tmp/svcspec_HxXApQ
Get service b5d47110-e534-46e3-915d-071bcbee5a80
Update service b5d47110-e534-46e3-915d-071bcbee5a80; spec: /tmp/svcspec_yGFJrp
Get service 8a94eb91-4096-48b9-b52d-20e030e9d063
Update service 8a94eb91-4096-48b9-b52d-20e030e9d063; spec: /tmp/svcspec_VlitkR
Get service 61bbcc34-9762-4517-8e57-f13ce25906b8_kv
Update service 61bbcc34-9762-4517-8e57-f13ce25906b8_kv; spec: /tmp/svcspec_Ym7LU3
Get service 72798453-18c0-4558-9bfe-bd2cd281621f
Update service 72798453-18c0-4558-9bfe-bd2cd281621f; spec: /tmp/svcspec_MEXsl8
Get service a64b10d5-4a3f-4d5d-b370-a03e17887cc1_com.vmware.vsan.health
Don't update service a64b10d5-4a3f-4d5d-b370-a03e17887cc1_com.vmware.vsan.health
Get service f32bcd6e-a903-4abe-b86d-09b37f1b6611
Update service f32bcd6e-a903-4abe-b86d-09b37f1b6611; spec: /tmp/svcspec_aT5C2O
Get service 209bfaf2-5e74-469d-bd14-f8bce3c67431
Update service 209bfaf2-5e74-469d-bd14-f8bce3c67431; spec: /tmp/svcspec_6O_TjH
Get service 8d954074-5555-4676-946c-add534bd22f1
Update service 8d954074-5555-4676-946c-add534bd22f1; spec: /tmp/svcspec_YGBFci
Updated 26 service(s)
Status : 100% Completed [All tasks completed successfully]

With this we have successfully configured VMCA as Subordinate CA of internal Microsoft CA and replaced SSL certificates of All vCenter services.

27. Now open the URL of vCenter server to see if it has valid certificate. You can notice the Lock icon of SSL certificate of URL, also see the issued By and Issued to details.

28. See the certificate Chain, you should see the Certificates of your Root CA in chain.

Now login to vCenter web client and make sure all Services are running fine.

Also you can login to vCenter web client with Administrator@vsphere.local account and go to Certificate Authority to view root Certificate, In Use certs, Issued or revoked certificates.

In my next post, I will share some common errors which we may get while replacing SSL certificates.

 

Thanks…!