For more than a decade, VMware presents vSphere-UI to help users to fulfill many important functionalities including exporting log bundles. There are multiple ways to collect the log bundles for vCenter. One of them is vSphere-UI workflow that provides with multiple user-friendly screens. Users have gained ability through the UI to collect system logs, host logs, VM logs individually whichever they want to download in their system.
This blog explains downloading steps in UI, helps in understanding all key/optional features. We will explore few common errors that are found in vsphere_client_virgo log and their probable explanation.
Downloading Steps
Export VC Logs: There are two workflows to download log bundles for vCenter.
- Workflow I -> is from VC action menu option – ‘Export System Logs…’.
User opens the popup window for ‘Export System Logs…’ by clicking on the specific action menu item on vCenter.
The window has two steps to start generating log bundle. In first step, information about hosts and an option to include VC logs in log bundle have been mentioned. User can either select host(s) or include only vCenter logs or both at a time to proceed to next step. Each host contains manifest driven information of system log files. In next step, union of all manifests are calculated to show them in a tree-based structure. Specifically, to export only vCenter logs, user can opt out selection of host(s). If no host is selected, manifest tree won’t be displayed in next step.
Will discuss the next step which is common with the step to download ESXi/VM log bundle.
- Workflow II -> is ‘Export Support Bundle’ of a selected vCenter under “Administration -> Deployment -> System Configuration”.
User can choose among three options clicking on ‘Export Support Bundle’:
- Full support bundle – it consists of information on all services.
- Minimal support bundle – it provides information about basic product and configuration.
- Custom support bundle – user has been given to select information to include provided in manifest files.
Export ESXi/VM Logs: User can separately download log bundle for single ESXi/VM by clicking on same action menu item on respective object.
It opens a single page modal which is similar with second step of VC log export window. User can select manifest data from the tree, set performance data and save a password for encrypted core dumps. The manifest tree data may vary from object to object. Information about these fields is elaborated in another sections.
Final step: After all selections are done, clicking on “Export Log” button closes the popup and starts generating bundle.
It first generates a zip name which is stored in session and opens a new tab in browser. The zip name (aka ‘downloadId’) as shown below is used for further communication with server after bundle is generated.
|
1 2 3 |
private void downloadLog(final HttpServletRequest request, final HttpServletResponse response, @PathVariable("downloadId") String downloadId, boolean... isSystemConfig) {…} |
Once bundle generation is complete, a system folder popup appears in browser where user must select a location to start downloading the log bundle.
User can save the log bundle in the system. It will start downloading the generated bundle and gets saved once completed. During this downloading step, if any failure happens (discussed in error elaboration section), downloading gets interrupted and the downloaded file will be corrupted.
What are manifests?
The manifest files are the key configuration/setup files which the vm-support tool refers during execution. The manifest files explain the nature of the commands the tool has to execute, the logs that are to be collected, time out value while executing a command, and error handling if a command or log collection has encountered an error.
These manifest files are located under /etc/vmware/vm-support and have the extension of mfx and read while collecting logs:
|
1 2 3 4 |
def _readManifestFiles(self): """ Create Manifest objects from manifest files. Search for prolog and epilog manifests. (Determined by name and group).To keep things simple we only deal with one prolog and one epilog file. We read _all_ manifest files no matter whether an mfx is disabled or the user wants to exclude it. |
A typical manifest file will have the following fields that explain its nature:
- Manifest name: System – The manifest name assigns a name to the manifest file that can be used across multiple manifest files. The manifest file is unique and is identified by the file name to which it has been created.
- Manifest group: Logs – Group tells the vm-support tool that this manifest file is categorized under a particular group and the file needs to be called when the vm-support tool is executed by a Group name.
- Manifest default: Enabled – This field ensures whether this manifest file is indeed authorized to be executed during run-time of the support tool. If a manifest file is set to Disabled, then the support tool ignores the manifest file and moves on to the next one. While running vm-support, without any additional command line options, the vm-support tool executes the manifest files that are set to be enabled.
- Manifest tag: base – The tag field assists the user to categorize multiple manifest files while running the vm-support tool based on the tag fields. Multiple tags can be provided to a manifest file which allows the same manifest file to be used by the engineer to have an idea of the components that it comprises.
- Action Options file/command – The last field contains the actual set of commands to be executed and the corresponding logs to be collected for diagnosis. The vm-support tool provides the user pre-defined commands that can be used to trigger system commands or to gather any logs.
Now let’s understand few checklists which are optional but important ones that might be very useful if included while generating log bundles. Besides selecting optional manifests from tree users have options to pass these parameters such as performance data and passwords for encrypted core dumps.
Performance Data Collection
To capture the performance data user can set duration (by default 300) and interval (5 seconds) which means that it will capture data for logs for 300 seconds with after an interval of every 5 seconds.
Why Core Dumps Encryption is required?
A host support bundle may include core dumps which may include some customer data. When vSphere Virtual Machine Encryption is present and active, a core dump may include cryptographic information (For example: passwords, cryptographic keys). The captured data is both necessary and unavoidable. It’s required to find and fix the reported issue.
When vSphere Virtual Machine Encryption is present and active, core dumps will be encrypted. This provides protection for any ESXi host data captured by a core dump. If the encrypting key is unknown, the core dump and its data are unreadable.
Some Known Errors found in Virgo Log and their explanation
Users might have commonly seen below errors in vsphere_client_virgo log resulting in log bundle download failure.
[ERROR] http-nio-5090-exec-25 c.v.vsphere.client.h5.admin.controllers.ExportLogsController Error while downloading logs. org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
In downloading steps, last and most crucial part is to select a system folder through window pop-up. If user cancels intentionally or delays to do the action or failed to do due to connectivity, above error is thrown and makes a mark in virgo log.
[ERROR] http-nio-5090-exec-25 c.v.vsphere.client.h5.admin.controllers.ExportLogsController Generate log bundles task returned null
[ERROR] http-nio-5090-exec-5 c.v.vsphere.client.h5.admin.controllers.ExportLogsController Error while downloading logs. java.lang.IllegalArgumentException: Invalid guid, no matching VcService found
The reason behind this might be anything such as session invalidation / connectivity issue / the VC is down which restricts to get the VC instance.
[ERROR] http-nio-5090-exec-1 c.v.vsphere.client.h5.admin.controllers.ExportLogsController Error while downloading logs. java.io.IOException: Server returned HTTP response code: 409 for URL: https://<<vc_ip>>/cgi-bin/vm-support.cgi?manifests=
It causes when user opens ‘Export system log’ window through VC action menu when already a generated log bundle is in process of download in same environment. The ESX does not support invoking multiple times the /cgi-bin/vm-support.cgi endpoint.
That’s all, we hope this article helped you to understand the basic things about export procedure. The vSphere UI team is looking for your feedback and ideas which you can provide in the comments below.
