Completely Disable Time Synchronization for your VM

Some administrative practices, like a bad habit, have more lives than the proverbial cat – they tend to stay around forever. It is, therefore, very comforting when one finds a problematic administrative practice that has not just been universally abandoned by administrators, but is also on the top of any junior administrator’s “configurations sure to get you dis-invited from the next user group meetup” list.

Take the case of the old practice of synchronizing a virtual machine’s clock with its host’s clock in a vSphere environment. That used to be “the thing to do” way back when. It was actually the default configuration option on the ESX platform in those days. Until everyone got wiser and the message went out to every admin far and wide that such configurations was no longer kosher. Even VMware got religious and stopped making that option the default behavior.

Now, if you are like one of the hundreds of vSphere administrators with whom I have interacted over the past years, you are probably unaware of the fact that, even when the “Synchronize clock with ESXi host” option is disabled (unchecked) on the VMware Tools properties of the virtual machine, the virtual machine will still occasionally synchronize its clock with the ESXi host. Oh, you knew that? Show-off 🙂 You probably did because VMware actually documented the behavior in a rather oft-overlooked knowledge base article – Disabling Time Synchronization

As mentioned in that KB, the following vSphere operations can cause the clock on a virtual machine to be reset (backward or forward) to match the clock on the host on which it resides.

Consider the following scenarios:

• Windows Active Directory Domain Services
  Windows Active Directory provides a loose sync time management infrastructure for all machines joined to the AD Domain. This time-keeping capability is sufficient and reliable enough for Windows and AD’s operations, particularly in ensuring that Kerberos authentication succeeds in the Forest/Domain and in-guest applications that do not require strict time accuracy (in sub-seconds). All Windows machines that are joined to a particular Windows Domain default to using the Windows-provided w32time service which looks at the domain/forest time hierarchy for its authoritative time synchronization. The PDC Emulator (PDCe) at the root of the forest is considered the time synchronization Marshall in the Forest, and it is usually configured to synchronize its time with a reliable, external clock and propagate time down the hierarchy throughout out the Forest. If the PDCe VM were migrated to the ESXi host with the bad clock, that operation will impact the accuracy and reliability of the PDCe VM’s clock. The resultant impact will be felt throughout the Active Directory infrastructure.
• Windows Applications
  Although Microsoft applications do not typically require strict time accuracy, most of these applications have time skew thresholds beyond which they begin to manifest failures and outages. For example, clustered Windows application nodes, or applications like Exchange Servers cannot function reliably if their time become severely divergent. Consequently, the clustered workloads will become unavailable, users will be unable to authenticate and access resources, etc
• Oracle Applications
  Time synchronization between Oracle RAC cluster nodes is crucial. While a deviating time between the servers in a cluster does not necessarily lead to instability, asynchronous times can make it harder to manage the cluster as a whole. One reason is that timestamps are written using the local node time. Log analysis can be impacted severely if the times in a cluster deviate significantly.The Oracle Clusterware requires the same time zone environment variable setting on all cluster nodes which is used for all processes e.g databases, Oracle ASM, and any other managed processes.
  A central time server in the data center, accessed by NTP, is typically used to synchronize the server times to prevent deviating times between the cluster nodes.It is best to avoid sudden time adjustments on individual nodes, which can lead to node evictions when performed too abruptly.Recommendation is to Disable Time Synchronization for all Oracle VMs irrespective of the kind of Oracle Application they host.

For enterprises with existing large number of virtual machines, where manually implementing this configuration changes is not a trivial undertaking, VMware has released a VMware vRealize Orchestrator (vRO) Workflow Package to assist in the effort. The workflow can be started from the VMware vRealize Orchestrator (vRO) client. If the vRO is already registered in a vCenter instance, then the workflow can also be initiated directly from the vSphere Web Client. In either case, you first need to import the attached workflow package  into the vSphere environment (using the vRO Client) before you can successfully use it in your vSphere environment.

The Package is available for Download Here

• NOTE: Microsoft introduced the “Secure Time Seeding (STS)” feature in Windows Server 2016, primarily in an attempt to detect and correct time synchronization errors, using outgoing SSL connections, and to proactively mitigate such occurences. It has since been proven that STS error-detection and correction logic is sufficiently flawed as to have become problematic and very disruptive in the enterprise. Microsoft has, therefore, been recommending that Customers consciously disable this feature in an Active Directory infrastructure where reliable and optimal time synchronization can be accurately provided the Windows NTP or other time-keeping facilities.
  • For completeness, here is the instructions for disabling STS in Windows (Note: this is better controlled through a GPO setting in the Active Directory infrastructure:Registry Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
    Value Name: UtilizeSslTimeData
    Value Type: REG_DWORD
    Value: 0

    Command Line: reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\w32time\Config /v UtilizeSslTimeData /t REG_DWORD /d 0 /f