Security Scanners and VCF

(Day 6 of 22 in our Cybersecurity Awareness Month series. Today: security scanners)

Security scanners are a reality of modern IT. Auditors ask for the reports, compliance frameworks require the scans, and security teams build dashboards from the results. Scanners are good tools, and they find real problems. They also produce a lot of findings that turn out not to be problems at all. That gap causes a lot of friction. A security team sees a long red list. An infrastructure team knows the environment is fully patched. An auditor wants a clean report, and nobody wants to argue.

In our post about vulnerabilities we noted that severity is not risk. Scanners are where that idea is most evident, because a scanner reports what it can see, and it cannot see context. Today we’ll look at how scanners work, why their findings for VMware Cloud Foundation (VCF) need context, what not to do about it, and what to do instead.

How Security Scanners Work

“Scanner” covers several different tools, and they see very different things.

Network scanners: These probe a system from the outside, over the network. They look at open ports, service banners, TLS settings, and version numbers, then match what they find against a list of known CVEs. They never log in.

Authenticated scanners: These log in to the system, over SSH or through an API, and list the installed packages and settings. They see more detail, and they need more access to get it. There is often tension here with the use of SSH, because we strongly recommend not enabling SSH in VCF except for troubleshooting. We’ll discuss this below.

Agent-based scanners: These install software on the system itself, and that software reports back continuously. These aren’t common in the context of VCF because installing software on VCF component appliances isn’t supported.

Image and SBOM scanners: These read a software bill of materials (SBOM), or the contents of an installer or container image, and match the component versions against CVE lists. They never touch a running system. These are increasingly common for container images.

Compliance scanners: These check configuration settings against a baseline, like a DISA STIG, PCI DSS, or even our Security Configuration Guide. They answer a different question. A vulnerability scanner asks whether the system has a known flaw, and a compliance scanner asks whether the system matches the baseline.

Most of these tools work the same way underneath. They find a version number and look it up in a list to see if that version has a vulnerability. That approach is fast, and it scales to thousands of systems, but it has limits.

Why Scanners Need Context for VCF

VCF breaks several of the assumptions that scanners routinely make, and this raises questions.

ESX Is Not Linux

ESX is its own operating system. It is neither Linux nor UNIX, even though parts of it look familiar. A scanner that treats ESX as Linux applies Linux checks, and the results may not apply to ESX. File permissions are a common example, because ESX does not use the Linux permission model.

Appliances Are Not General-Purpose Servers

vCenter and the other VCF appliances run on Photon OS, but they are purpose-built systems, similar to the way ESX is. A good way to think of them is the same way you think of the firmware on a network switch or a storage array. We deliver the appliance as an atomic unit, and we support and maintain it that way. As the label on a home appliance like a refrigerator often says, “no user-serviceable parts inside,” but we understand why organizations want to scan them anyhow: it’s about risk. We support organizations knowing what is on their network.

Consider who is logging in to the appliances. For VCF, the people with access to the appliances are already VCF administrators. In our post about vulnerabilities we talked about attack paths and vectors, and how important it is to consider those when assessing risk. This is true on the appliances, too. Suppose a scanner flags a vulnerable software package. If the only people who can reach it already have full, explicit access to VCF itself, the risk is likely lower than the finding suggests.

Version Numbers Are Not Enough

Many scanner checks are a simple comparison of a package version against a list. That’s a reasonable approach, but it doesn’t capture context or configuration. For example, a vendor can remediate a vulnerability by backporting the fix to the older version of the software package. This will happen at times when moving to the new version is too disruptive because of larger upstream changes. In these cases, scanners won’t know if the fix is present.

In other cases, the vulnerability lies in a part of a software package that we don’t include, or we don’t configure. The scanner won’t know that, and it will report a problem. Scanners tend to assume the worst case, and, understandably, many scanners report the highest score they can find. They would rather report too much than miss something that a threat actor could take advantage of.

No View of Your Environment

A scanner does not know that your management interfaces sit on an isolated network. It does not know who can log in, or what your firewall rules allow. Those things also help decide your risk, especially in the context of a vulnerability’s attack vector. As we mentioned earlier, the risk changes considerably if the vulnerability exists in a part of the stack that only an already-trusted administrator can get to.

Scanner settings matter here, too. Most scanners have a sensitivity setting. At the highest settings they report things they could not confirm, which adds length to the report without adding certainty.

Artificial Intelligence and Vulnerability Volume

There is always a natural tension between upstream software projects and those who build things atop them. It might be an open-source project and a downstream service, or an OS support team and an application support team inside an enterprise. The upstream project keeps moving, and the downstream work is often focused on stability and predictability for the application running on it. This is where the preference for leaving a working system alone comes from (“if it ain’t broke don’t fix it”), and, frankly, it is a reasonable one.

That approach is harder to sustain nowadays. Large language models (LLMs) are very good at finding vulnerabilities in source code, and the volume of CVEs has risen sharply as a result. For example, some software projects, like the Linux kernel itself, sometimes see hundreds of CVEs filed against them per day.

Downstream software needs to choose a point in time to build and, more importantly, test against. With the upstream project moving exceptionally fast because of the vulnerabilities, gaps form between what is shipped and what a scanner sees. Are those trouble? Usually not, but to help, VCF 9.1 ships updates more frequently, and along with those releases is information you can use to check.

CVE Disposition Information

The Broadcom Support Portal has the answer to “how can you check on a particular CVE?” We ship “CVE Disposition” information as part of the VMware Cloud Foundation software releases. The data is in a CSV file in the downloads, under the “Additional Downloads” tab (search for “CVE”). Broadcom Knowledge Base (KB) article 450429 outlines how to get it, and what the review covers:

https://knowledge.broadcom.com/external/article/450429/locating-and-downloading-cve-status-list.html

What Not to Do With Scanners

When a report has a lot of findings, it is tempting to clear them quickly. Some of the ways to do that make things worse.

If you can avoid it, don’t enable SSH for the scanner. SSH and the ESX Shell are troubleshooting interfaces. They are off by default, and the Security Configuration Guide recommends leaving them off, and leaving the host in lockdown mode. A scanner that logs in via SSH inherently has privileged access, because the shell has no way to limit what it can reach. You would be weakening the system in order to measure it. The scanner and its operators are now in scope for audits, too. Many scanners offer an authenticated API scan, which can allow them to interact with the VCF stack in a controlled, logged, and managed way.

Don’t install agents on ESX or the appliances. These are closed systems, and unauthorized software changes to them are not supported. An agent is also one more thing to patch, audit, and protect, on the systems that matter most.

Don’t fix findings by hand. Replacing a package inside an appliance with one from another Linux distribution is not supported, and it tends to break things in ways that are hard to diagnose. Fixes arrive through product updates.

Don’t aim an untested scanner at production. VCF generally handles routine network scans without trouble, but scanners vary, and so do their settings. Try the scanner in a test environment first, then on a few hosts in maintenance mode.

Don’t turn the sensitivity all the way up. You get a longer report and less certainty about what is in it. KB 450429 has guidance at the end about individual scanner settings.

Tactics That Work With Scanners

You can still get a thorough, authenticated scan without any of that.

Use the API, not the shell. Many scanners already support this. Scanners can read versions, installed components, and configuration through the same vSphere API that your own tools use. That gives the scanner real data to work from, and SSH stays off.

Give the scanner its own account. Create a dedicated service account with a custom read-only role, holding only the privileges the scanner’s documentation lists. Don’t hand it an administrator account. A dedicated account also makes the scanner’s activity easy to find in your logs.

Scan hosts through VMware vCenter. Many scanners can discover and assess ESX hosts through vCenter. The scanner then needs a network path to one place, not to every host. This also fits well with ESX lockdown mode.

Confirm the scan authenticated. A scan that cannot log in often falls back to what it can see from the outside, and those findings are less precise. Look in the results for confirmation that the authenticated checks ran.

Verify certificates. Scanners offer to skip certificate checks when they meet a self-signed certificate. Trusted certificates make that unnecessary.

Check the scanner’s support list. Confirm that the scanner covers the version you run before you trust its findings. Scanner content can lag behind new releases.

Treat shell-only checks as exceptions. Some compliance checks still expect shell access. Provide other evidence for those, such as the audit commands in the Security Configuration Guide, and record the exception.

Collect only what you need. Inventory options, like listing every virtual machine, add load and put more of your data in the scanner. Turn off what you don’t use.

What to Do With Findings From Scanners

None of this means you can ignore the report. Some findings are real, and the point about chaining from our post about vulnerabilities applies here, too. The goal is to sort the findings quickly and fix the ones that are real.

1. Get current first. Many findings disappear once you apply the latest updates, and a fully patched environment has far fewer findings to explain.
2. Get to VMware Cloud Foundation or VMware vSphere Foundation 9.1 or newer. There are many new security improvements in 9.1, new tools that make patching and security operations easier, and more frequent patch releases to help you avoid many of these situations.
3. Check for a VMSA. If we have published an advisory for the CVE, it tells you which versions are affected and which have the fix.
4. Check the CVE disposition list, as noted above. We publish our review of scanner-reported CVEs as a CSV file in the Broadcom Support Portal, under “Additional Downloads” for your VCF release.
5. Find out how the scanner decided. Ask your scanning team for the logic behind the check. If it is a version comparison, compare it against what is installed.
6. Apply context. Ask the questions from our post about vulnerabilities. Can an attacker reach it? What do they need first?
7. Write it down. If a finding does not apply, record why, with the evidence. Your auditor needs that more than they need a clean report.

Scanners and Your Workloads

Everything above is about scanning the infrastructure. Scanning workloads is different. Guest operating systems are general-purpose systems, which is exactly what scanners are built for. Authenticated scans and agents make more sense there, and the findings are usually what they appear to be.

VCF helps on this side, too. You can clone a workload and scan the clone, or take a snapshot before you remediate a finding, so a bad fix is easy to undo. VCF is built to help mitigate risk. Use those capabilities.

Frequently Asked Questions About Scanners

Q1: Which scanner does Broadcom recommend?

A1: We don’t recommend specific scanners. We do suggest choosing one that lists VCF or vSphere as a supported target. A scanner that understands the platform makes fewer inaccurate assumptions about it.

Q2: My auditor wants an authenticated scan of ESX over SSH. What do I tell them?

A2: Offer them an authenticated scan through the API instead. Many scanners support it, and it gives the auditor real data without a shell. SSH on ESX is a troubleshooting interface that is off by default, and turning it on for a scanner weakens the system being audited. It can help to compare ESX to a storage array or a network switch, where nobody expects a shell for the scanner. For the few checks that still expect one, offer other evidence, such as the audit commands in the Security Configuration Guide.

Q3: The scanner flagged a package inside an appliance. Can I update that package myself?

A3: No. Changing packages inside an appliance is not supported, and it can break the appliance. The fix arrives in a product update, which is the supported way to get it.

Q4: I applied the update, and the scanner still shows the finding. Why?

A4: Check three things. First, confirm the update finished and the version is what you expect. Second, update the scanner’s own definitions and scan again. Third, check the CVE disposition data to see if it is resolved, and how.

Q5: Is it safe to run network scans against VCF?

A5: Routine network scans are generally fine. Test a new scanner or a new scan profile first, as described above. Keep in mind that a scanner needs network access to your management interfaces, and that access deserves the same care as any other.

ESX can run inside a VM (“nested ESX,” for testing only), so if you want to test scanning, this is a safe way to do it. Same for vCenter.

Q6: What about compliance scanners?

A6: Check that the scanner has content for the version you run. Security guidance for one major version does not fit another, and a scanner that applies VCF 5.2 checks to VCF 9.1 gives you wrong answers in both directions.

Scanners are worth having and are a powerful tool for finding problems that might form in an environment. They are most useful when the people reading the reports know what the scanner can and cannot see, and where to go for more information.