Optimizing Horizon Blast for High-End 3D graphics with NVIDIA graphics cards

One of my customers is currently doing a POC for Horizon 7.3.1 – The customer owns various companies which deliver single parts for their end-product. In all of these there is a design department where new parts or machines to produces them are designed. There are a lot of graphical workstations used overall – but they are spread out in the region where the customer resides. The ultimate target for this customer is, to ease management for all these workstations and to centralize this workload. – A typical VDI use case and perfectly suited for Horizon together with the powerful NVIDA graphics cards such as the NVIDIA Tesla M60. – The customer is all Windows – and the current standard desktop OS is still Windows 7 (64-Bit)…

The customer got all the servers and graphics cards to build the 3D Horizon ESXi host. Horizon 7.3.1 was installed and they started testing. – The first result was not really exciting, performance at the designer’s desk was slow, or for some use case even unusable. – This was before we did any optimization.

How does Horizon Client and Blast protocol work, and what can be done? – The answer is pretty easy.

First of all we have to know that VMware delivers the Horizon Client and also the agent in a default mode that suits most of the use cases. This means performance is average, bandwidth consumption is average. It works pretty well for the standard user, but it needs optimization for low bandwidth environments – a use case which is described in VMware documentation, and it needs adjustment for end graphics applications like 3D software. In my case the customer used CATIA 3D, but it will need adjustment for any other high end graphics application, where people are doing complex construction design.

Horizon Client 4.6.1

Did you know that there is a network setting  for Blast protocol in the Horizon client user interface that can impact your performance? Horizon client has 3 different built-in profiles:

  1. Excellent
  2. Typical (default)
  3. Poor

If you have your average client PC and your average user using Office type apps on a LAN environment go for typical. If you don’t have enough bandwidth or users accessing over the WAN take poor. – But if you need high end performance and if you are connected in a LAN take Excellent.

So my customer’s designers are not strictly sitting in the LAN of the headquarter – but they are something like 50 km around the headquarter and they have good (LAN-type 1GB) connection to the headquarter – that’s fine and works well with the excellent setting.

H.264 is enabled by default. – H.264 is very well suited for videos and it works for 3D as well. The advantage of H.264 to day is, that most of the modern PCs (or their graphics cards) have a H.264 graphics encoder – and Horizon client will use this hardware decoder if present. – That’s a performance gain. H.264 on the other hand is doing compression which is not 100% lossless, so for certain use cases we might have to switch that – but in our case it did not have a big impact and it was acceptable. – The really bad effects can be seen if you write red text on a blue background – if that is your favorite setting in which app so ever H.264 will not be your friend, and you have to deselect it.

To be able to configure Blast you have to disconnect your client from the connection server. Then you have to open the 3 bars menu on the left and select Configure VMware Blast.

Then just select the first Radio button “Excellent” – this was the Horizon Client part.

Having done this “one-click” change already changed a lot for the customer. – They were able to move their 3D objects around the screen at 90% satisfaction ratio. We did see some blurriness and movement was not always smooth.

The initial setup of the virtual Windows 7 CAD desktop was pretty much reduced. The customer allocated only 2 vCPU for this virtual desktop, which is far below VMware’s recommendation. – Analyzing the CPU behavior of the virtual desktop we saw CPU going up to well over 90% – bad sign. During testing we moved the pretty complex pretty fast over the screen, and CPU hit 100% – a reason why it was a bit jumpy.. – But we are in the virtual world. Just add another CPU Core to the system – yes you have been reading correctly only 1 vCPU was added, so we had a configuration of 3 vCPUs (1 socket 3 cores).  – So we did not succeed to bypass 70%. – That sounds like a good configuration.  We are using CATIA 3D on Windows 7-64 with 3 vCPUs only, on a Windows 7 image that has been optimized using the VMware OS Optimization Tool. https://labs.vmware.com/flings/vmware-os-optimization-tool

Horizon agent 7.3.1

VMware delivers Group Policy templates which can be copied to a domain controller in your active Directory environment. There is a special VMware Blast ADMX file amongst all these GPO templates. You can basically configure everything you need to get you Horizon environment customized to your needs by using these GPO templates. We will have a closer look at the blast settings:

There are three main settings we had to adjust to get it really smooth when moving around, and remove some blurriness during the movement.

The ultimate smoothness delivered:

MaxBandwidthKbpsPerMegaPixelSlope = 25000
Specifies the maximum bandwidth, in kilobits per second (kbps), for a VMware Blast session. The bandwidth includes all imaging, audio, virtual channel, USB, and VMware Blast control traffic. The default is 1 Gbps.

Max Frame Rate = 60
Specifies the maximum rate of screen updates. Use this setting to manage the average bandwidth that users consume. The default is 30 updates per second.

The blurriness disappeared:

H.264maxQP = 28
H.264minQP = 10
Specifies the image quality for the remote display configured to use H.264 encoding. You can specify the minimum and maximum quantization values that determine how much an image is controlled for lossless compression. You can specify a minimum quantization value for the best image quality. You can specify a maximum quantization value for the lowest image quality. You can specify the following settings:

  • H264maxQP (available range of values: 0-51, default: 36)

  • H264minQP (available range of values: 0-51, default: 10)

So this was our final setting:

I would like to thank Kiran Rao Director of Horizon Product Management, and Hilko Latinga EUC architect and co-author of the NVIDIA whitepaper below. Both helped me with these parameters and made our POC a success. Although these parameters are documented it is not really obvious which parameter has to be changed to get the desired result.

Reference documentation:

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-horizon-7-view-blast-extreme-display-protocol.pdf

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-horizon-view-graphics-acceleration-deployment.pdf

https://docs.vmware.com/en/VMware-Horizon-7/7.3/horizon-remote-desktop-features/GUID-220442CF-EA01-470E-A381-1BED9BC0B81C.html

Posted in End User Computing, vSphere | 8 Comments

Windows vCenter 5.5 / 6.0 Migration to vCenter Appliance (VCSA) 6.5 U1

It is time to think about upgrading “older” vSphere environment to the latest version 6.5 U1. The journey to the cloud, VSAN and lots of other new possibilities as well as the approaching end of support date for vSphere 5.5 (2018/09/19) are more than enough reasons to start planning a vSphere Upgrade.

VMware recently announced the official end of Windows vCenter: https://blogs.vmware.com/vsphere/2017/08/farewell-vcenter-server-windows.html

Traditionally a lot of customers are still using vCenter on Windows. VMware’s longer term strategy is targeted to the modern vCenter Server Appliance (VCSA). This Photon OS based version of vCenter offers very good performance and as it is an appliance les complexity. All configuration and management task can easily be fulfilled using either the HTML management interface of the standard vSphere WebClients (the traditional Flex client as well as the HTML5 Web client). – So to make the story short – it is time to replace the Windows vCenter  with the VCSA.

vSphere 6.5 or better the vCenter Appliance Installer has an integrated vCenter migration assistant which allows to migrate a Windows vCenter to a brand new vSphere 6.5 U1 vCenter Appliance.

This is a pretty easy task, which consists of two parts.

  1. Start the migration tool on your old Windows vCenter. – This automatically launches a Pre-Check. You current Windows vCenter is analyzed to verify if the migration has access to all resources needed – and if the current version can be migrated correctly.
  2. Launching the VCSA 6.5 U1 installer, and selection the Migration Option. – This will create a brand new vCenter Server Appliance, migrate all the data we want to migrate, then switch of the old Windows vCenter and change the personality of the new vCenter Appliance to match the old Windows vCenter.

This is pretty cool because:

  1. Your brand new Appliance will have the same name and IP as your old vCenter, so all users, scripts and other surrounding modules should work without any change. – Shure – before doing your upgrade you have to double check if these surrounding components, like plug-ins, Backup solutions, agentless antivirus solutions etc. will be compatible with vCenter 6.5 – and yes you might need to update some of these modules first to guarantee a error free migration, but that’s your personal pre-check before you start the vCenter migration.
  2. If something goes wrong and the new appliance should not be operational – you still have your Windows vCenter which is untouched, you just need to reboot it….

I won’t write too much about how this migration works, there are lots of colleague bloggers offering click thru migration samples. There is also a pretty complete official blog about this upgrade and the migration. So before jumping into the upgrade check these resources:

General information about vSphere 6.5 U1:
https://blogs.vmware.com/vsphere/2017/07/vsphere-6-5-update-1-hood.html

Upgrade Considerations Part 1:
https://blogs.vmware.com/vsphere/2017/05/vsphere-6-5-upgrade-considerations-part-1.html

Upgrade Considerations Part 2:
https://blogs.vmware.com/vsphere/2017/07/vsphere-6-5-upgrade-considerations-part-2.html

Upgrade Considerations Part 2:
https://blogs.vmware.com/vsphere/2017/08/vsphere-6-5-upgrade-considerations-part-3.html

vCenter Migration Assistant Walkthrough:
https://blogs.vmware.com/vsphere/2017/01/vcenter-server-appliance-6-5-migration-walkthrough.html

 

Errors in the Migration Assistant of Windows vCenter 5.5 6.0 to vCenter Appliance 6.5 U1

The Migration assistant in my lab produced the following error running it for the first time (and yep these are pretty typical):

You have to run the following command repeatedly if you have to fix any errors..

Most errors have a pretty good description in their log files. The following is just a sample of two errors that could occur which can be fixed quickly. SOme items cannot be rtansfered automatically, such as COntent Libraries stored directly on the Windows vCenter Server or old plugins that won’t work with vCenter 6.5.

Here is a sample run of the assistant:

D:\migration-assistant\VMware-Migration-Assistant.exe

Initializing Migration Assistant...

Enter Administrator@vsphere.local password: *************

Your vCenter Server service is configured to run under the 'LAB\svc-vcenter' service account.

Enter 'LAB\svc-vcenter' service account password: ***********

==============================================================

DO NOT CLOSE this console until the migration is complete.

==============================================================

Extracting Migration Assistant scripts...

Running Prechecks...

Prechecks reported the following errors/warnings:

Error: User running the upgrade does not have 'Replace a process level token' privilege.

Resolution: Assign 'Replace a process level token' privilege to the user.

Error: VMware Update Manager health status is red Cannot proceed with migration

Resolution: Please check health status file at C:\Program Files (x86)\VMware\Infrastructure\Update Manager\docroot\vci\downloads\health.xml and follow make sure that VMware Update Manager health status is green before migration

Migration Assistant log files have been zipped at: C:\Users\administrator.LAB\Desktop\VMware-MA-logs-20170421085022.zip

Failed to initialize Migration Assistant. Resolve the above error(s), and relaunch Migration Assistant.

  1. Error: My vCenter server service runs with a custom service account. This is something that can happen at customer’s installations. Usually these service accounts use the least privileges necessary to run the service successfully. As the Migration assistant uses the service account user to migrate, it will have to launch some privileged actions. – So in my case I will have to change the Windows local policy to allow the user to replace a process token. Open Local Computer Policy > Computer Settings > Windows Settings > Security Settings > Local Policies > User Rights Assignment > “Replace a process level token” and add the vCenter Service User to the list of privileged users.
  2.  Error: Update Manager Service User? – In my lab Update Manager was using Local System User. As the script has to migrate the Update Manager service as well, the easisest way to fix this is – use the same user as for vCenter service – so I just reconfigured vCenter Update Manager Service to use the svc-vCenter account (same as above). – Restart service and retry the migration assistant.

I hope this gives you an idea what can happen. Just as an idea why not start the migration assistant before the actual migration and fix things before you start? This will give you less pressure on the actual day of migration.

What should you migrate? Does it make sense to migrate Events and Task information? If your vCenter database is really large this can take hours. Just migrating the important configuration information will speed up the process and give you a small vCenter database to begin with your new vCenter. If you want to migrate task and event data, it is maybe a wise idea to delete really old events and task data some days before you start the migration – by changing the vCenter settings for task cleanup and event cleanup (duration and activation).

 

Posted in vSphere | Leave a comment

vSphere Homelab Version 2017

Well there are a lot of people having a lot of home lab installations out there. The latest trending home lab servers are Intel NUCs. I started to think about replacing my old PCs I still use as physical ESXi servers. Looking at Intel NUCs is for sure an option. The price is really interesting, but the max mem is 32 GB and they come with one NIC only, and you cannot really put a lot of disk space into these beasts. – So my plan was to have a home lab with some VSAN capacity, and the option to extend them in the next 12 months. – So I started to look for an alternative to these NUC babies.

What I found was Shuttle XPC cube barebone. The latest model the SZ170R8V2 comes with 2 built-in Intel 1 GB NICs, four DDR4 DIMM slots offering 64 GB max memory, 4 x 3,5″ disk chassis and M.2 SSD card slot, and on top of all this 2 PCI slots all of this in a nice 33 x 21 x 19 cm cube and it accepts Intel Gen 7 CPU (with the latest BIOS). – The best of the Shuttle motherboard is it uses Intel standard chips for NIC and graphics, so just run ESXi 6.5 without any tweaking of drivers etc.

For sure it is not supported to run VSAN, but it works – and that’s the main point for a home lab! (the Shuttle has HDMI and display port video connectors, so maybe you need an adapter to connect you DVI or VGA monitor…)

Here is the link to the baby: SZ170R8V2

 

My configuration to start would be an I7 Gen processor (quad core) 32 GB memory, M.2 250 GB SSD as cache and a 512 GB SSD for capacity. This confiurationg can be extended  with maybe a dual 10 GB NIC, another 32 GB of memory and some SSD (there is space for 3 more SATA devices…) for capacity in the next months when prices hopefully drop…

So what was the price for this? – Living in Switzerland I just take the swiss price and convert it in US$ to give you an idea

1 x Shuttle Barebone SZ170R8V2

306 US$

1 x Corsair Vengeance LPX (2x, 16GB, DDR4-2400, DIMM 288) 

281 US$

1 x INTEL Core i7-7700 Kaby Lake Prozessor

299 US$

1 x SAMSUNG SSD 2.5″ 512GB SATA 6Gb/s 850 Pro

235 US$

1 x SSD 960 Evo M.2 250 GB

126 US$

1 x USB Stick 32 GB Kingston DTSE9 G2 (boot device)

  24 US$

TOTAL

1271 US$

 

The other good thing I saw so far, the Shuttle does not produce much heat, and it is pretty quiet!

Posted in VSAN, vSphere | 4 Comments

I hope you know RVTOOLS

I just noticed that there is a new version of RVTOOLS out. Since years I do use and recommend RVTOOLS to my customers. – I know there are tons of scripting guru’s out there who will get some of the information out of vSphere infrastructure. But RVTOOLS is just the one and only tool every vSphere admin can use with just one click, and everything you ever wanted to know about your vSphere infrastructure will be intelligently grouped in the tool, and by another click you can export all this information into an Excel spreadsheet.

So if you are not yet a user of RVTOOLS go and grab it now! You can download it for free, and I think it is more than worth of giving a donation to the developper of the greatest tool you can find out there!

http://www.robware.net/rvtools/

 

 

Posted in vSphere | Leave a comment

vSAN on vSphere 6.0 and vSphere 6.5 – Upgrade path – there are dependencies..

A few days ago VMware released the new version of vSphere 6.0 Update 3. – As you would expect this version also contains some awesome patches for vSAN, which will make it more stable and more performant. – see the ESXi 6.0 U3 Release notes! Great stuff. – But take care there are dependencies. As there are currently two release tracks of vSphere (6.0 and 6.5) which are updated regularly – there are slight differences in vSAN versions and sometimes you cannot easily upgrade from a vSphere 6.0 version to a 6.5 version if you are using the latest vSAN versions of 6.0. – Sounds confusing? – Yes it is somehow.

And it is right now. If you are going to update your vSphere 6.0 to U3 today  – then you will have to wait for vSphere 6.5 U1 if you want to upgrade to 6.5. – and a possible release date for 6.5 U1 is somewhen in summer or autumn? – This means if you plan to upgrade to 6.5 before summer, you should not update to 6.0 U3 today.

vSphere 6.0                 vSAN 6.0                      released March 2015

vSphere 6.0 U1            vSAN 6.1                      released September 2015

vSphere 6.0 U2           vSAN 6.2                      released March 2016

vSphere 6.5                  vSAN 6.5                      released November 2016

vSphere 6.0 U3           vSAN 6.2+patches  released 24 february 2017 **

** If you want to  update to vSphere 6.5 you will have to wait for 6.5 U1

I hope this helps for your upgrade planning!

Posted in VSAN, vSphere | Leave a comment

Some Performance hints for VSAN 6.2

With VSAN 6.2 VMware added a lot of new features, like dedup, compression and checksums. Some of these features can use CPU resources to a certain extent – depending on your workload.

In my personal experience with customers, I have found that the Checksum option can produce some overhead – especially in All-Flash configurations with extremly high IO, or if you are doing benchmarks (which also produces high IO load normally…)

If you are using all-SSD configuration and you can live without checksums, just change your VSAN policies accordingly. Use the “Disable object checksum” option in your storage policy and set the option to “YES”.

Here is the link to VMware’s VSAN documentation.

Are you using a Hybrid VSAN 6.2 configuration?

There is another hint for you. Since VSAN 6.2 VSAN is always calculation Deduplication / Compression data. – WHich in case of Hybrid configuration deos not make any sense at all. – But it produces a certain overhead.

There is a very important Knowledge Base article for you if you are using VSAN in hybrid mode. – Change some options and get better performance!

http://kb.vmware.com/kb/2146267

 

 

Posted in VSAN | Leave a comment

vCenter / SSO / PSC 5.1 5.5 and 6.0 mixed versions compatibility

You have multiple vCenters in different versions and want to upgrade to vCenter 6.0 (U2 or later)? – What is supported and what is unsupported?

There are some possible configurations where you can have different versions of SSO compared to your vCenter versions.  This automatically happens during upgrades. There are some combinations that are supported and some combination which are unsupported.

Especially re-installing or re-pointing and “old” vCenter to a newer SSO or PSC version is something which is only supported for vCenters 5.1 pointing to SSO 5.5 – the newer versions can only be installed or repointed to the identical versions.

The following kb contains the table with supported versions of vCenter and SSO/PSC:

http://kb.vmware.com/kb/2120504

Remember when updating your exsiting vSphere 5.5. SSO, you domain name is still “vsphere.local”. Although PSC supports self defined domain names, in an upgrade scenario you will always end with “vsphere.local” as changing the domain name is not supported.

This information can be found in the following kb:

http://kb.vmware.com/kb/2113115

In the above kb you will find a link to the documentation, showing different transitional scenarios and describing the functionality for the various states. See: ”Mixed-Version Transitional Environments in vCenter Server for Windows Upgrades” in the ”vSphere Upgrade Guide“.

Can I mix Windows PSCs with Appliance PSCs? – Yes it is allowed to have Windows PSCs and Appliance PSCs in the same PSC domain – but it is NOT allowed to create HA pairs with different OS PSCs. (also found in the above kb).

By the way if you are dealing with multiple vCenters, there are ways to create command-line installations or upgrades. For details see here:

Scripted way to install/upgrade VC 6.0u2 on Windows: www.vmware.com/…and-upgrade.pdf

Scripted way to deploy/upgrade vCSA 6.0u2: www.vmware.com/…and-upgrade.pdf

 

Posted in vSphere | Leave a comment

vCloud Networking and Security (VCNS) to NSX for vShield Endpoint Upgrade

Are you using Shield Endpoint to manage your offloading anti-virus solution? It is time now to update your vShield Manager to NSX Manager. vShield Network and Security (VCNS) is end of support September 2016. Your “old” vShield Endpoint manager is actually part of VCNS. You should start your upgrade asap, to have a supported solution in the future. NSX will be used to manage your vShield Endpoint environment. You don’t need any additional NSX licenses if you are using NSX for vShield Endpoint – you only need a new tool to manage your vShield Endpoint management.

It is not too hard to upgrade. There is the vCloud Network and Security to NSX Upgrade Guide which will guide you step by step thru the whole process:

http://pubs.vmware.com/NSX-62/topic/com.vmware.nsx.upgrade.endpoint.doc/GUID-EC5B912F-0E12-4032-9ED0-113DE5C6850C.html

https://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_upgrade_endpoint.pdf

Updated: Aug 29 2016

Before upgrading, please consult the NSX 6.2.4 Release Notes available from the NSX Documentation Center

http://pubs.vmware.com/Release_Notes/en/nsx/6.2.4/releasenotes_nsx_vsphere_624.html 

and http://kb.vmware.com/kb/2144295 – minimum recommended version for NSX for vSphere with GID, ESXi, and vCenter Server. – for Guest introspection make sure to use tools 10.0.9 – or later…

For VCNS End Of Life see: http://kb.vmware.com/kb/2144733

For supported partner Antivirus solutions (Trend Micro, Symantec, BitDefender, McAfee…) see http://kb.vmware.com/kb/2110078

Posted in End User Computing | 1 Comment

How to keep VMware TOOLS up-to-date?

Update VMware Tools?

You might have heard in autumn 2015 that VMware is de-coupling VMware Tools from virtualisation solutions. This means VMware Tools will be released separately and the latest VMware Tools will be compatible with all platforms, this means the latest VMware Tools will work with Workstation, Fusion, and all ESXi versions (5.X, and 6.X).

This is especially useful if you are in an upgrade scenario, with rolling upgrades – so you can update VMware Tools even before you upgrade the your first ESXi hosts…

The downside of this new strategy is, that having a new ESXi version (or an update version) will NOT give you the guarantee to have the latest VMware Tools version.

How does this work with VMware Tools?? – Just 4 easy steps!

VMware does not deliver a fully automated way to get the latest tools – VMware Update Manager is not yet able to download and update the VMware Tools source of the ESXi hosts.

But it is not too much work to deliver the latest Tools to existing ESXi hosts, which again will allow to follow the known Update options for VMware Tools in a vSphere environment.

The important thing to know is, how does ESXi host “know” that installed tools are “old”, and where are the tools taken from, when doing automatic updates?

That’s pretty easy: By default, ESXi includes VMware Tools under the /productLocker folder (which is actually a Symlink to the location on it’s local storage..)

Instead of copying the latest Tools version to all ESXi hosts, there is an advanced parameter, to point all hosts to a repository on shared storage.

  • Create a directory on a shared datastore (preferably called productLocker)
  • get the latest “VMware Tools” from the “Drivers & Tools” section of your ESXi download page: https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/6_0#drivers_tools
  • Copy the content of the latest VMware Tools ZIP or TAR archive to the directory on your shared datastore (content of productLocker must be the two directories “floppies” and “vmtools” containing the latest versions.
  • Change the advanced Setting on every ESXi host: UserVars.ProductLockerLocation to pointing to the shared storage path (example value: /vmfs/volumes/ShareDatastoreName/productLocker) – note that the value is CASE SENSITIVE!!! – ShareDatastoreName is the name of your shared datastore  – This location will also be used by Update Manager when you prefer to use Update Manager for automatic VM updates…..

As soon as the host reboots – the new location will be vaild.

There is a powershell script – which is not supported by VMware – which will automate the whole process see: http://www.vtagion.com/automate-vmware-tools-shared-product-locker-configuration/ – you can either use the script as is or just copy the parts you need to automate for your environment…

Thanks to Brian Graf for the above information!

It is even possible to make the path change without rebooting – this requires the change of the symbolic link on the ESXi filesystem.

(details can be found on Brian Graf’s Blog post if needed https://blogs.vmware.com/vsphere/2015/09/vmware-tools-lifecycle-why-tools-can-drive-you-crazy-and-how-to-avoid-it.html

The details of the above steps can be found on this blog post: https://blogs.vmware.com/vsphere/2015/11/updating-to-vmware-tools-10-must-read.html BUT DO NOT USE THE TOOLS Download link of this blog page as it will lead you to Tools 10.0.0 !!) –

I hope this information will help you to keep your VMware Tools in a healthy and updated stage.

Posted in vSphere | 2 Comments

Change OS (update from Windows 2008 to Windows 2012 R2) of vCenter 5.5 – Change vCenter 5.5 SSO architecture, virtualize a physical vCenter 5.5

1. Summary

The following checklist describes the process of migration (update) the vCenter 5.5 OS (for example from Windows 2008 to Windows 2012 R2) by actually re-installing a fresh vCenter 5.5 – using the exsisting vCenter database, inventory database and certificates. A following step could be to update this “refreshed” vCenter 5.5 to vCenter 6.0. – And yes I am talking about Windows vCenter to Windows vCenter I am not talking about vCenter appliance.

I had various questions from customers running vCenter 5.5 on outdated OS, or still running vCenter on a physical server. That’s why I collected the steps to get to a redesigned vCenter which will be seen just as the “same” from all surrounding apps like AV or backup etc.

This upgrade will be done in five different phases. This ensures a minimal production impact and various fallback possibilities at different points in time. Fallback is always possible.

CAUTION: If you are running additional services on your vCenter server always double check compatibility with your target system. – For example vSphere Authentication Proxy only supports IIS 6 and IIS 7 – so it would not be compatible with Windows 2012 or later. (http://kb.vmware.com/kb/2058689) – always check Product Interoperability matrix and Solution/Database Interoperability first before updating! http://partnerweb.vmware.com/comp_guide2/sim/interop_matrix.php?

The five migration phases are:

  1. Build a new virtual server running Windows 2012 R2 using the existing vCenter and inventory databases. – This step will be done during the first maintenance window. (During this upgrade, all virtual servers and ESXi hosts will continue to run without interruption.) – Only vCenter actions such as create/delete VMs or taking snapshots etc. will not be possible. High availability of virtual machines is NOT affected. Duration of this maintenance window is about 8 hours.
  2. Step two, which will be at a later point in time, will be the actual upgrade of the refreshed vCenter from version 5.5 to version 6.X. For this step, there will be another maintenance window of about 4 hours. Otherwise the same access and availability situation as for step one
  3. Step three will be upgrade of ESXi hosts. This can be a rolling update – cluster by cluster. The current configuration allows upgrade during normal operation times, as there is always room for a single ESXi failure (or maintenance.
  4. Step four will be upgrade of VMware Tools for the virtual machines. All VMware Tools should be upgraded to Version 10.0.5 or higher. – This step can be combined with Windows or Linux patching cycles and it can be done at any time – even before the upgrade of vCenter!
  5. Last step will be the upgrade of virtual machine hardware. This has to be done after VMware Tools upgrade and after the ESXi host upgrade. This step requires virtual machine reboot and therefore can be combined with regular guest OS patching.

2. Upgrade Flow

2.1       Prerequisites for vCenter Migration/Upgrade

The following prerequisites should be met before the upgrade:

 

  1. passwords:
Domain Administrator
Local Administrator account old vCenter Server
Local Administrator account new vCenter Server
Old vCenter administrator@vsphere.local
New vCenter administrator@vsphere.local
old vCenter SQL sa
New vCenter SQL sa

 

  1. Remote Access (DRAC/ILO) to old physical vCenter Server
  2. Installation sources for Windows Server 2012
  3. Installation sources for MS SQL Server 2014
  4. Installation sources for vCenter 5.5
  5. Installation sources for vCenter 6.0 Update X
  6. Rvtools installed on a Management station

3. Phase 1 – Migration physical vCenter 5.5 to virtual Windows Server 2012 R2 – or migrate vCenter 5.5 to another Windows OS

3.1       Preparation

  1. Create a new virtual Windows 2012 R2 server with latest patches and latest VMware Tools.
    1. Temporary name
    2. NOT joined to domain
    3. .NET 3.5 Feature has to be installed
  2. SQL Server should not be installed on the same VM as vCenter server. This is only acceptable for very small or LAB environments – details see vCenter Installation documentation. For larger environments have an existing SQL server ready to host vCenter db or install a dedicated SQL server for vCenter.

3.2       Migration

These are the steps for the actual migration. – On the “old” vCenter do the following:

  1. If necessary disable DRS on the cluster where the old vCenter Server
    is running
  2. Create an inventory with rvtools on your management server
  3. Stop and deactivate all vCenter Services
  4. Backup vCenter certificate on a central backup store
    C:\ProgramData\VMware\VMware VirtualCenter\SSL.
  5. Backup vCenter WebClient certificate on a central backup store
    C:\ProgramData\VMware\vSphere Web Client\ssl
  6. Backup vCenter SQL database and copy file to central backup store (and write down permissions for vc db (SQL Server should not be installed on the same VM as vCenter server. This is only acceptable for very small or LAB environments – details see vCenter Installation documentation)
  7. Write down ODBC configuration
  8. On the source machine, stop the Inventory Service (if still running)
  9. On the source machine, open the command prompt and change the directory to
    vCenter_Server_installation_directory\Infrastructure\Inventory Service\scripts
  10. Run the following command to back up the Inventory Service database.
    bat -file backup_file_nameWhen the backup operation finishes, the message Backup completed successfully appears. Copy file to central backup store
  11. Copy the SSL certificate from source path C:\ProgramData\VMware\Infrastructure\Inventory Service\ssl\
    to your backup store
  12. Export all Customization Specifications to the central backup Store
  13. Note ESXi host where the newly created Windows server resides
  14. Shutdown the “old” vCenter

On the newly created virtual vCenter server do the following:

  1. Shutdown server
  2. Create snapshot (connect to ESXi host where this virtual machine resides)
  3. Delete Active Directory account of physical vCenter server
  4. Start and login to the new virtual server, change server name (same as “old” vCenter Server)
  5. Change IP address of the NEW server (same as “old” vCenter Server)
  6. Add server to domain
  7. Installation of SQL Server 2014
    Management Tools, Client Tools & Backwards Compatibility (SQL Server should not be installed on the same VM as vCenter server. This is only acceptable for very small or LAB environments – details see vCenter Installation documentation) (if SQL is installed on a separate server just install SQL Native Client on vCenter Server
  8. Start SQL Agent and configure AUTOMATIC start
  9. Restore vCenter db and set permissions
  10. Create vCenter ODBC Connectors
  11. Create new Update Manager db
  12. Create vCenter ODBC Connector
  13. Create path and Restore vCenter certificate
    C:\ProgramData\VMware\VMware VirtualCenter\SSL.
  14. Create path and Restore vSphere Web Client certificate
    C:\ProgramData\VMware\vSphere Web Client\ssl
  15. Create path and restore Inventory certificate
    C:\ProgramData\VMware\Infrastructure\Inventory Service\ssl\
  16. Start custom install for vCenter – then install new SSO service
  17. Install vCenter 5.5 WebClient
  18. Install vCenter 5.5 Inventory service
  19. Install vCenter 5.5 Service using the restored vCenter database
    (accept database Upgrade if you are using latest VC 5.5 Update)
  20. Install Update Manager
  21. Install vCenter Client
  22. Stop the Inventory Service.
  23. Open the command prompt and change the directory to
    vCenter Server install location\Infrastructure\Inventory Service\scripts.
  24. Run the following command to restore the Inventory Service database.
    restore -backup backup_file_name
    When the restore operation finishes, the message The Restore completed successfully message appears.
  25. Restart the inventory service
  26. Login to vCenter and check functionality
  27. Import all Customization Specifications to the central backup Store
  28. Restart vCenter VM
  29. Check functionality
  30. If all is ok – remove vCenter Server snapshot
  31. If necessary enable DRS on the cluster where the vCenter Server
    is running

3.2.1     Fallback

In case something happened:

  1. Shutdown virtual vCenter Server
  2. delete vCenter domain account
  3. if SQL Server was on a separate server – restore old vCenter database
  4. Start the old physical vCenter Server
  5. Add server to workgroup – add server to domain
  6. Activate all vCenter services
  7. Restart server

4. Phase 2 Upgrade the virtual vCenters to vSphere 6

4.1       Upgrade

  1. Findout on which host vCenter is running
  2. Connect latest vCenter 6 ISO to vCenter
  3. Shutdown vCenter
  4. Start vSphere Client and direct connect to host where vCenter runs
  5. Create Snapshots of vCenter
  6. Start vCenter
  7. Start update process for vCenter 6
  8. Check functionality
  9. restart vCenter Servers
  10. check functionality
  11. if successful delete snapshot

4.1.1     Fallback Scenario

  1. Find out on which host vCenter is running
  2. Start vSphere Client and direct connect to host where vCenter runs
  3. Shutdown vCenter
  4. Revert to snapshot
  5. Re-start vCenter
  6. check functionality

5. Phase 3 – Upgrade ESXi Hosts

Before the upgrade of an ESXi cluster, we should move a test-VM to this cluster and configure it to the clusters “local” network(s). This will help us to do a ping test before migrating production VMs to a new host.

5.1       Host Upgrade

  1. Set one ESXi Host to maintenance mode
  2. Set DRS to manual
  3. Shutdown ESXi Host
  4. Update BIOS and Firmware of all adapters
  5. Update ESXi to version 6.X
  6. Update ESXi to latest patch version
  7. Install storage plugins if necessary (NetApp Plugin)
  8. Release host from maintenance mode
  9. Migrate test VM to new host
  10. Do basic connectivity test of migrated VM
  11. Set DRS to automatic
  12. Repeat procedure for next ESXi host in cluster
  13. If all hosts are updated set DRS back to automatic

5.1.1     Fallback

  1. Re install ESXi host with ESXi 5.5
  2. Configure host and add to cluster

6. Phase 4 – Upgrade VMware Tools

VMware Tools inside the virtual machines have to be updated to the latest version (10.0.5 or higher). – This update is not dependent of any other update. It can happen any time. This step requires reboot of the updated VM. The latest tools are supported by vSphere 5.5 as well as vSphere 6.

VMware tools can either be updated using Update Manager or via software deployment products.

7. Phase 5 – Update virtual hardware

As soon as the VMware Tools are updated and the host I running ESXi 6 virtual hardware can be upgraded to the latest level.

This again requires a reboot of the virtual machine.

Posted in vSphere | Leave a comment