Skyline Cloud Support Experience

Skyline Advisor Pro Proactive Findings – September Edition

VMware Skyline Advisor Pro releases new proactive Findings every month. Findings are prioritized by trending issues in VMware Support, issues raised through post escalation review, security vulnerabilities, and issues raised from VMware engineering, and customers.

For the month of September, we released 42 new Findings. Of these, there are 18 Findings based on trending issues, 6 based on VMSAs, and 18 based on post escalation reviews. Skyline Advisor Pro now has over 1,000 active findings. We picked a few Findings from each of these categories which stand out in this release.

VMware Support Trending Issues

VMware Support trending issues are KBs that have solved a large number of SRs and/or viewed a large number of times.

In KB 89131, ESXi Host fails with PSOD “NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS” PSOD will show functions FastSlabAllocSlow, Pkt_AllocHandleWithSize & vmk_PktAlloc in the backtrace. The root cause is memory pressure on the host plus some locking issues that occur every time the slab increases. Each time the slab is increased to the next level, it allocates 2MB. This allocation increases reservation and may hit kernel-side locking, so it takes longer to allocate pages due to fragmentation. This behavior was not observed before 7.0u2. This issue is resolved in VMware ESXi 7.0 Update 3g (build number 20328353).

  • ID: vSphere-PSODFastSlabAllocSlow-KB#89131
    • Description: ‘FastSlabAllocSlow’, ESXi 7.0 host fails with a PSOD ‘NMI IPI: Panic requested by another PCPU. RIPOFF(base):RBP:CS’.
    • Resolution: This issue is resolved in VMware ESXi 7.0 Update 3g (build number 20328353).
    • Helpful Link: https://kb.vmware.com/s/article/89131
    • Severity: CRITICAL

Post Escalation Review

VMware Technical Support has developed a Post Escalation Review process. We review critical escalations which come into our Escalation Management team and determine steps to prevent these escalations in the future with other customers. One of the outcomes of this process is the creation of Skyline Findings.

In KB 87986, vpxd service crashes when its unable to connect to ESXi hosts with vpxuser credentials. The vpxuser credentials for each ESXi host are refreshed at a normal interval. Hashes of these passwords are also stored within vCenter. When vCenter is reverted to a snapshot, it can cause some of the rotating vpxuser passwords that it stores to also be reverted to a previous version. vCenter is no longer able to contact the ESXi hosts with these credentials and will occasionally encounter a race condition that results in the vpxd service crashing and creating a coredump. This issue is resolved in VMware vCenter Server 7.0 Update 3e (build number 19717403).

  • Finding ID: vSphere-vpxdservicecrashes-KB#87986
    • Description: The vpxd service crashes when its unable to connect to ESXi hosts with vpxuser credentials.
    • Recommendations: This issue is resolved in VMware vCenter Server 7.0 Update 3e (build number 19717403).
    • Helpful Link: https://kb.vmware.com/s/article/87986
    • Severity: MODERATE

Security Vulnerabilities

The Skyline Advisor Pro team added support for Horizon Agent versions in this release. This allowed for the creation of 4 Skyline Findings based on Horizon Agent-based VMSAs over the last few years. Below is a list of the Horizon Agent-based VMSAs we included in this release.

To review all released Findings for the month of September and all current active VMware Skyline Findings please go to the VMware Skyline Findings Catalog.

Most Viewed Findings in Last 30 Days

Below are the most viewed Findings by users in Skyline Advisor Pro.

  1. vSphere-VCFEsxNTPRule-KB#81647
  2. vSphere-CustomizeWindowsGuests-KB#1020716
  3. vSphere-VCFEsxRemoteSysLogRule-KB#81648
  4. vSphere-VMmorethan3snapshots-KB#1025279
  5. vSphere-XHCI-USB-controller-VMSA#202204
  6. vSphere-VMsnapshotover7days-KB#1025279
  7. vSphere-SpectreMeltdown-VMSA#201804-2
  8. vSphere-Vmtoolsmemoryleak-KB#76163
  9. vSphere-VmUnresponsivememoryleak-KB#2077302
  10. vSphere-L1TerminalFault-VMSA#201820-3
  11. vSphere-Log4jremotecodeexe-VMSA#202128
  12. vSphere-CVE-2020-3992-VMSA#202023
  13. vSphere-CVE-2022-22943-VMSA#202207
  14. vSphere-EsxiBuildInconsistent
  15. vSphere-VCFVMMonitoring-KB#81636
  16. vSphere-PSODQLogicFCHBA-KB#83790
  17. vSphere-VCFNoDVS-KB#81639
  18. vSphere-EndpointCertExpiration
  19. vSphere-VCSALogDiskSpace-KB#76563
  20. vSphere-CVE-2021-21997-VMSA#202111

Comments

Leave a Reply

Your email address will not be published.