Monday, July 27, 2020

How to Backup and Restore Shielded VM Local Certifcates


One of the hot new technologies in Hyper-V 2016 is Shielded Virtual Machines. This feature plugs a few long-standing security holes in the hypervisor space that were exacerbated by the rise of hosting providers. It’s ridiculously easy to start using Shielded Virtual Machines, but its simplicity can mask some very serious consequences if the environment and guests are not properly managed. To make matters worse, the current documentation on this feature is sparse and reads more like marketing brochures than technical material.

The material that does exist implies that Shielded Virtual Machines require a complicated Host Guardian Service configuration and a cluster or two. This is not true. You can use Shielded Virtual Machines on standalone hosts without ever even finding any setup for Host Guardian Service (HGS). Using a properly configured HGS is better, but it is not required. Standalone mode is possible. “Standalone” can apply to non-domain-joined hosts and domain-joined hosts that are not members of a cluster. I did verify that I could enable VM shielding on a non-domain-joined host, but I did not, and will not, investigate it any further. This article will discuss using Shielded Virtual Machines on a domain-joined Hyper-V host that is not a member of a cluster and is not governed by a Host Guardian Service.

What are Shielded Virtual Machines?

A Shielded Virtual Machine is protected against tampering. There are several facets to this protection.

Unauthorized Hosts Cannot Start Shielded Virtual Machines

Only systems specifically authorized to operate a Shielded Virtual Machine will be able to start it. Others will receive an error message that isn’t perfectly obvious, but should be decipherable with a bit of thought. The primary error is “The key protector could not be unwrapped. Details are included in the HostGuardianService-Client event log.” The details of the error will be different depending on your overall configuration.

No Starting Shielded VMs on Unauthorized Hosts
No Starting Shielded VMs on Unauthorized Hosts

This feature is most useful when combined with the next.

Unauthorized Hosts Cannot Mount Virtual Hard Disks from Shielded Virtual Machines

The virtual hard disks for a Shielded Virtual Machine cannot be opened or mounted on unauthorized systems. Take care as the error message on an unauthorized host is not nearly as clear as the message that you receive when trying to start a Shielded Virtual Machine on an unauthorized host, and it could be mistaken for a corrupted VHD: “Couldn’t Mount File. The disk image isn’t initialized, or contains partitions that aren’t recognizable, or contains volumes that haven’t been assigned drive letters. Please use the Disk Management snap-in to make sure that the disk, partitions, and volumes are in a usable state.”.

For small businesses, this is the primary benefit of using Shielded Virtual Machines. If your VM’s files are ever stolen, the thieves will need more than that.

VMConnect.exe Cannot be Used on a Shielded Virtual Machine

Even administrators can’t use VMConnect.exe to connect to a Shielded Virtual Machine. In case you didn’t already know, “VMConnect.exe” is a separate executable that Hyper-V Manager and Failover Cluster Manager both call upon when you instruct them to connect to the console of a virtual machine. This connection refusal provides a small level of protection against snooping by a service provider’s employees, but does more against other tenants that might inadvertently have been granted a few too many privileges on the host. Attempting to connect results in a message that “You cannot connect to a shielded virtual machine using a Virtual Machine Connection. Use a Remote Desktop Connection instead.”

No VMConnect for Shielded VMs
No VMConnect for Shielded VMs

The upshot of the VMConnect restriction is that if you create VMs from scratch and immediately set them to be shielded, you’d better have some method in mind of installing an OS without using the console at all (as in, completely unattended WDS).

What are the Requirements for Shielded Virtual Machines?

The requirements for using Shielded Virtual Machines are:

  • Generation 2 virtual machines

That’s it. You’ll read a lot about the need for clusters and services and conditional branches where a physical Trusted Platform Module (TPM) can be used or when administrator sign-off will do and all other sorts of things, but all of those are in regards to Guarded Fabric and involve the Host Guardian Service. Again, HGS is a very good thing to have, and would certainly give you a more resilient and easily managed Shielded Virtual Machine environment, but none of that is required. The only thing that you must absolutely have is a Generation 2 virtual machine. Generation 1 VMs cannot be shielded.

Generation 1 virtual machines can be encrypted by Hyper-V. That’s a topic for another article.

How Does the Shielded Virtual Machine Mechanism Work on a Standalone System?

Do not skip this section just because it might have some dry technical details! Ignorance on this topic could easily leave you with virtual machines whose data you cannot access! Imagine a situation in which you have a single, non-clustered host with a guest on a Scale Out File Server cluster and you enable the Shielded VM feature. Since all of the virtual machine’s data is on an automatically backed-up storage location, you don’t bother doing anything special for backup. One day, your Hyper-V host spontaneously combusts. You buy a new host and import the VM directly from the SOFS cluster, only to learn that you can’t turn it on. What can you do!? You could try crying or drinking or cursing or sacrificing a rubber chicken or anything else that makes you feel better, but nothing that you do short of cracking the virtual machine’s encryption will get any of that data back. If you don’t want that to be you, pay attention to this section.

Shielded Virtual Machines are Locked with Digital Keys

Access to and control of a Shielded Virtual Machine is governed by asymmetric public/private encryption keys. In a single host environment without a configured Host Guardian Service, these keys are created automatically immediately after you set the first virtual machine to be shielded. You can see these certificates in two ways.

The CERTUTIL.EXE program is available on any system, including those without a GUI. At an elevated command prompt, type:

You’ll be presented with a dialog that shows the Shielded VM Encryption Certificate. Click More Choices and it will expand to show that certificate and the Shielded VM Signing Certificate:

VM Shielding Certificates
VM Shielding Certificates

You can click either of the certificates in the bottom half of the dialog and it will update the information in the top half of the dialog. Click the Click here to view certificate properties link, and you’ll be rewarded with the Certificate Details dialog:

Certificate Details
Certificate Details

This dialog should look fairly familiar if you’ve ever looked at a certificate in Internet Explorer or in the Certificates MMC snap-in. We’ll turn to that snap-in next.

The Microsoft Management Console (MMC.EXE) has a dependency on the Explorer rendering engine, so it is only available on GUI systems. You can use it to connect to systems without a GUI, though, as long as they are in the same or a trusting domain.

  1. After that, you’ll need to indicate which computer to control. In my example, I want the local computer so I’ll leave that selection. You can connect to any computer in the same or a trusting domain, provided that the user account that you started MMC.EXE with has administrative privileges on that computer:
    Choose Local or Remote Computer
    Choose Local or Remote Computer
  2. After you OK out of all of the above dialogs, MMC.EXE will populate with the certificate tree of the targeted computer account. Expand Shielded VM Local Certificates, then click the Certificates node. If you have shielded a virtual machine, you’ll see two certificates:
    VM Shielding Certificates in MMC
    VM Shielding Certificates in MMC

You can open these certificates to view them.

Not to put too fine a point on it, but these two certificates are absolutely vital. If they are lost, any virtual machine that they were used to shield is also permanently lost… unless you have the ability to crack 2048-bit SHA256 encryption. There is no backdoor. There is no plan “B”.

If you are backing up your host’s operating system using traditional backup applications, a standard System State backup will include the certificate store.

If you are not backing up the management operating system, then you need a copy of these keys. I’ll give you directions, but the one thing that you must absolutely not miss is the bit about exporting the private keys. The shielding certificates are completely useless without their private keys!

Using CERTUTIL.EXE is the fastest and safest way to export certificates.

  1. Open an elevated command prompt.
  2. VM Shielded Certificates with Serial Numbers
    VM Shielded Certificates with Serial Numbers
  3. Use the mouse to highlight the first serial number, which should be for the encryption certificate, then press [Enter] to copy it to the clipboard.
  4. To export the VM shielding encryption certificate, type the following, replacing my information with yours. Use right-click to paste the serial number when you come to that point: certutil -exportPFX -p "2Easy2Guess!" "Shielded VM Local Certificates" 169d0cacaea2a396428b62f77545682e c:\temp\SVHV02-VMEncryption.pfx
  5. Use the mouse to highlight the second serial number, which should be for the signing certificate, then press [Enter] to copy it to the clipboard.
  6. To export the VM shielding signing certificate, type the following, replacing my information with yours. Use right-click to paste the serial number when you come to that point: certutil -exportPFX -p "2Easy2Guess!" "Shielded VM Local Certificates" 5d0cb1f0fa8b34b24e1195c41d997c19 c:\temp\SVHV02-VMSigning.pfx
  7. Ensure that the PFX files that you created are moved to a SAFE place and that the password is SECURED!

If you ever need to recover the certificates, use this template:

You’ll be prompted for the password on each one.

The MMC snap-in all but encourages you to do some very silly things, so I would recommend that you use the certutil instructions above instead. If you must use the UI:

  1. Open MMC and the Certificates snap-in using instructions from the “Viewing Shielded Virtual Machine Certificates Using the Certificates MMC Snap-In” section above.
  2. Highlight both certificates. Right-click them, hover over All Tasks, and click Export…
  • Click Next on the informational screen.
  • Leave the defaults on the Export File Format page. If you know what you’re doing, you can select Enable certificate privacy. Do not select the option to Delete the private key!! The host will no longer be able to open its own shielded VMs if you do that!
  • On the Security tab, you must choose to lock the exported certificate to a security principal or a password. It’s tempting to lock it down by security principal, and it might even work for you. I almost always use passwords because they’ll survive where security principals won’t. If you do choose this option, only use domain principals, use groups not names, use more than one, and make double-, triple-, and quadruple- certain that your Active Directory backups are in good shape. If you’re one of those types that likes to leave your Hyper-V hosts outside of the domain for whatever reason, the Groups or user names option is a good way to lose your shielded VMs forever.
  • The final screen is just a summary. Click Finish to complete the export.
  • Ensure that the PFX files that you created are moved to a SAFE place and that the password is SECURED (or if you used one or more security principals, hope that nothing ever happens to them)!

If you ever need to recover these certificates, I would again recommend using certutil.exe instead. The GUI still makes some dangerous suggestions and it takes much longer. If you insist on the GUI:

  1. Open MMC and the Certificates snap-in using instructions from the “Viewing Shielded Virtual Machine Certificates Using the Certificates MMC Snap-In” section above.
  2. Right-click in the center pane and hover over All Tasks, and click Import…
  3. Click Next on the introductory screen.
  4. On the File to Import screen, navigate to where your certificate backups are. Note that you’ll need to change the filter from *.cer, *.crt to *.pfx, *.p12 to see them.
  5. The Password part of the Private key protection screen is fairly easy to figure out (and won’t be necessary at all if you protected by security principal). Do make sure to check the Mark this key as exportable box. If you don’t, then you won’t be able to export the private key. It’s not strictly necessary, since you do have the file that you’re importing from. At least, you have it right now. Something could happen to it, and then you’d have no way to generate a new one.
    Import as Exportable
    Import as Exportable
  6. The final screen is just a summary. Click Finish to import the certificates.

Do take good care of these certificates. They are literally the keys to your Shielded Virtual Machines.

Why Does the Certificate Say “Untrusted Guardian”?

The consequence of not using a full Host Guardian Service is that there’s no independent control over these certificates. With HGS, there’s independent “attestation” that a host is allowed to run a particular virtual machine because the signature on the VM and the signing certificate will match up and, most importantly, the signing certificate was issued by someone else. In this case, the certificate is “self-signed”. You’ll see the term “self-signed” used often, and usually incorrectly. Most of the time, I see it used to refer to certificates that were signed by someone’s internal certificate authority, like their private domain’s Enterprise CA. That is not self-signed! A true self-signed certificate is signed and issued by a host that is not a valid certificate authority and is only used by that host. The most literal meaning of a self-signed certificate is: “I certify that this content was signed/encrypted by me because I say so.” There is no independent verification of any kind for a true self-signed certificate.

Can I Use Shielded VMs from an “Untrusted Guardian” on Another Hyper-V Host?

Yes. These virtual machines are not permanently matched to their source host. That’s a good thing, because otherwise you’d never be able to restore them after a host failure. All that you need to do is import the keys that were used to sign and encrypt those virtual machines on the new target host into its “Shielded VM Local Certificates” store, and it will then be able to immediately open those VMs. There will not be any conflict with any certificates that are already there. This should work for Live Migrations as well, although I only tested export/import.

If you like, you can unshield the VMs and then reshield them. That will shield the VMs under the keyset of the new target host.

What Happens When the Certificate Expires?

I didn’t test, so I don’t know. You could try it out by forcing your clock 10 years into the future.

Realistically, nothing bad will happen when the certificate expires. An expired certificate still matches perfectly to whatever it signed and/or encrypted, so I see no reason why the VMs wouldn’t still work. You can’t renew these certificates, though, so the host will no longer be able to use them to sign or encrypt new VMs. If this is still something that you’re concerned about 9 years and 11 months after shielding your first VM, be happy that your host made it that long and then unshield all of the VMs, delete the certificates, and reshield the VMs. New 10 year certificates will be automatically created and give you another decade to worry about the problem.

How Do I Know if a VM is Shielded?

The “easiest” way is the checkbox on the GUI tab. There’s also PowerShell:

Virtual hard drives are a bit tougher. Get-VHD, even on Server 2016, does not show anything about encryption. You can test it in a hex editor or something else that can poke at the actual bits, of course, but other than that I don’t know of a way to tell.

Are Shielded VMs a Good Idea on an Untrusted Host?

I’m not sure if there is a universal answer to this question. Without the Host Guardian Service being fully configured, there is a limit to the usefulness of Shielded VMs. I would say that if you have the ability to configure HGS, do that.

That said, shielding a VM on an untrusted host still protects its data if the files for the VM are ever copied to a system outside of your control. Just remember that anyone with administrative access to the host has access to the certificate. What you can do, if you’ve got an extremely solid protection plan, is export, delete, and re-import the certificate without marking the private key as exportable. That’s risky, because you’re then counting on never forgetting or losing that exported certificate. However, even a local admin won’t be able to steal virtual machines without having access to the exported key as well

No comments:

Post a Comment