Authors:
- Madhav Achari (Systems Test)
- Amitrajit Chatterjee (Staf II Solutions Architect)
VMware Cloud Foundation 4.0 on Dell EMC VxRail introduces support of workload domain deployment and operations utilizing the public API interface. In this multi-part blog series, we will provide guidance on how to use these new APIs to perform common deployments and operations. In Part 1 of the blog series, we covered the deployment of workload domains while Part 2 will cover various workload domain related operations.
Detailed API documentation is located at VMware {code} site. This can also be accessed from inside the SDDC Manager from the Developer Center.
Pre-requisites
Please complete VCF-VxRail management domain bring up and deployment of Workload Domain. The subsequent domain operations can be done via public api as detailed below.
Obtain Access token
The SDDC Manager APIs are secured using token-based authentication. As a first step before invocation of any API, an access token must be obtained. Invoking the Token API returns a token pair – access token and refresh token. To invoke an API, the access token must be passed in the Authorization header as a Bearer token.
SSH to the SDDC Manager vm or a Linux jump vm and execute the relevant curl commands
|
1 2 3 4 5 6 7 8 9 10 |
[root@LinJump home]# curl 'https://<sddc manager ip>/v1/tokens' -X POST -H 'Content-Type: application/json' -d '{"username": "[email protected]","password": "RandomPwd!123"}' --insecure | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 732 0 662 100 70 2340 247 --:--:-- --:--:-- --:--:-- 2339 { "accessToken" : "<long token>", "refreshToken" : { "id" : "f6340391-53c7-41cd-85f8-7b49acd1200f" } } |
Refresh Access token
If the Access token expires, a Refresh token will need to be obtained. A refresh token can be used any number of times to obtain an access token until it has not expired or been revoked. To check if any API call is failing due to expired token, review the log file /var/log/vmware/vcf/domainmanager/domainmanager.log. Sample error message is given below. Follow the API guidance to obtain the Refresh token.
|
1 2 3 |
ERROR [b69d45f72861c113,1a18] [c.v.v.s.t.service.TokenServiceImpl,http-nio-127.0.0.1-7200-exec-1] Unknown exception while parsing jwt token JWT expired at 2020-05-26T05:14:56Z. Current time: 2020-05-26T05:27:21Z, a difference of 745310 milliseconds. Allowed clock skew: 0 milliseconds. 2020-05-26T05:27:21.311+0000 ERROR [b69d45f72861c113,1a18] [c.v.v.s.a.AuthorizationFilter,http-nio-127.0.0.1-7200-exec-1] Error decoding JWT token com.vmware.vcf.security.token.service.error.TokenInvalidException: JWT expired at 2020-05-26T05:14:56Z. Current time: 2020-05-26T05:27:21Z, a difference of 745310 milliseconds. Allowed clock skew: 0 milliseconds. |
Add Host to Cluster
First image the new host and add it to the relevant VxRail cluster per Dell-EMC VxRail procedures. Multiple hosts can be added in simultaneously to a specific cluster.
Get Cluster ID
Next SSH to the SDDC Manager vm and execute the following command in preparation of completing the add host procedure to the cluster.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
[root@LinJump home]# curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <Token ID>' 'https://<sddc manager ip>/v1/clusters/' --insecure | json_pp { "elements" : [ { "isStretched" : false, "hosts" : [ { "id" : "a840d34a-1f16-411b-ae67-33a14f091836" }, { "id" : "feff6499-9046-478e-81bb-7e87e045873d" }, { "id" : "11d25843-481b-45b9-92b4-d76dce4cb8b7" }, { "id" : "64ac7a89-ad50-4074-a2d9-0a952f85b6ed" } ], "primaryDatastoreName" : "VxRail-Virtual-SAN-Datastore-3abfae4a-7693-4ba8-b5f8-d67b689ef072", "name" : "VxRail-Virtual-SAN-Cluster-3abfae4a-7693-4ba8-b5f8-d67b689ef072", "primaryDatastoreType" : "VSAN", "id" : "5de6dec9-c64a-4887-beae-2050c1bf27e0" }, { "isStretched" : false, "hosts" : [ { "id" : "0ed89841-7f51-4c38-baf3-c9e82c2e45af" }, { "id" : "59a5e207-5777-4536-8883-422f9b8901b8" }, { "id" : "bc8560d2-4a54-4287-90db-10d0e0a1d2d7" } ], "primaryDatastoreName" : "VxRail-Virtual-SAN-Datastore-9a78e0b9-b1e0-41c9-a78e-8f0851b04839", "name" : "VxRail-Virtual-SAN-Cluster", "primaryDatastoreType" : "VSAN", "id" : "bd852932-8938-4e73-a0c4-9e3b4548afa6" #cluster id } ] } |
Prepare Add Host ( Cluster Expansion ) Json
Create a json file with information related to the host being added in. It should match the number of hosts which were added to the VxRail cluster previously.
File Add_Host_API :
|
1 2 3 4 5 6 7 8 9 10 |
{ "clusterExpansionSpec" : { "hostSpecs" : [ { "ipAddress" : "192.168.15.58", "hostName" : "rack14-008.vxrail.local", "username" : "root", "password" : "RandomPwd!123" } ] } } |
Validate the Input Json
Validate the input json file created above to ensure there are no issues.
|
1 2 3 4 5 6 7 8 9 |
[root@LinJump home]# curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer <Token-ID>' 'https://<sddc manager ip>/v1/clusters/bd852932-8938-4e73-a0c4-9e3b4548afa6/validations' -d @Add_Host_API --insecure | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 326 0 127 100 199 285 446 --:--:-- --:--:-- --:--:-- 447 { "id" : "44bcb5ff-b0ae-4be3-b8b0-14c213325278", "executionStatus" : "IN_PROGRESS", "description" : "Validating Cluster Expansion Spec" } |
Validation of Input Json completed successfully
Wait for the validation of the input json to complete.
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@LinJump home]# curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <Token-ID>' 'https://<sddc manager ip>/v1/clusters/validations/44bcb5ff-b0ae-4be3-b8b0-14c213325278' -d @Add_Host_API --insecure | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 375 0 176 100 199 1760 1990 --:--:-- --:--:-- --:--:-- 2010 { "validationChecks" : [], "id" : "44bcb5ff-b0ae-4be3-b8b0-14c213325278", "resultStatus" : "SUCCEEDED", "executionStatus" : "COMPLETED", "description" : "VALIDATE_CLUSTER_EXPANSION_WORKFLOW" } |
Execute the Add Host (Cluster Expansion) Workflow
Once the validation completes successfully, execute the following command to kick off the add host workflow. Rerun the command to monitor the status of the tasks. It can also be monitored from the SDDC manager GUI.
|
1 2 3 4 5 6 7 8 9 10 |
[root@LinJump home]# curl -X PATCH -H 'Content-Type: application/json' -H 'Authorization: Bearer <Token-ID>' 'https://<sddc manager ip>/v1/clusters/bd852932-8938-4e73-a0c4-9e3b4548afa6/' -d @Add_Host_API --insecure | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 360 0 161 100 199 571 706 --:--:-- --:--:-- --:--:-- 708 { "status" : "IN_PROGRESS", "creationTimestamp" : "2020-05-25T11:01:19.436Z", "name" : "Adding new host(s) to vxrail cluster", "id" : "79dbee35-a229-4753-8b1c-855cb4cd2202" } |
Review the cluster details from the SDDC Manager to ensure the correct number of hosts show up.
Complete Succesful API Response is given below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 |
curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>' 'https://192.168.15.28/v1/tasks/79dbee35-a229-4753-8b1c-855cb4cd2202' -d @Add_Host_API --insecure | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5010 0 4811 100 199 53952 2231 --:--:-- --:--:-- --:--:-- 54056 { "resources" : [ { "resourceId" : "bd852932-8938-4e73-a0c4-9e3b4548afa6", "type" : "Cluster" }, { "resourceId" : "da05e14a-e13a-4178-b797-49a7d20cb49d", "type" : "Esxi" }, { "resourceId" : "01dc67ce-f59b-4701-86c2-f766469fefe1", "type" : "Domain" } ], "resolutionStatus" : "UNRESOLVED", "status" : "Successful", "creationTimestamp" : "2020-05-25T11:01:19.924Z", "name" : "Adding new host(s) to VxRail cluster VxRail-Virtual-SAN-Cluster", "isCancellable" : false, "subTasks" : [ { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:06:16.123Z", "name" : "Release Lock Action", "description" : "Release Lock for ESXi Host Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:06:15.301Z", "name" : "Add Host Update Status In Logical Inventory Action", "description" : "Update ESXi Host Data in Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:06:14.641Z", "name" : "Add Host Update Status In Logical Inventory Action", "description" : "Update ESXi Host Data in Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:06:13.718Z", "name" : "Update Known VxRail Availability Zone Hosts", "description" : "Update VxRail Hosts SSH Key To Known Hosts Configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:05:49.134Z", "name" : "Exit Maintenance Mode on ESXi Hosts", "description" : "Exit Maintenance Mode on ESXi Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:05:49.082Z", "name" : "Validate NSX-T Connectivity Action", "description" : "Validate NSX-T component connectivity after NSX-T network configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:45.240Z", "name" : "Configure NSX-T Transport Node Action", "description" : "Create NSX-T Data Center Transport Nodes from Discovered Nodes" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:44.876Z", "name" : "Detach Transport Node Collection Action", "description" : "Detach Transport node collection from cluster" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:44.503Z", "name" : "Prepare Add Host Network Transfer Input Action", "description" : "Gather existing NSX-T network configuration to join NSX-T Fabric" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:43.757Z", "name" : "NSX-T Add Host Header", "description" : "Gather input to add host to NSX-T Fabric and configure overlay networking on VDS." }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:43.707Z", "name" : "Validate NSX-T Connectivity Action", "description" : "Validate NSX-T component connectivity after NSX-T network configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:42.906Z", "name" : "Prepare NSX-T Info From Inventory Action", "description" : "Validate NSX-T component connectivity before starting NSX-T network configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:42.711Z", "name" : "Generate Nsxt Input For Add Host Action", "description" : "Get NSX-T input data" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:41.582Z", "name" : "Clear Alarms on ESXi Hosts", "description" : "Clear Alarms on ESXi Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:27.390Z", "name" : "Enter Maintenance Mode on ESXi Hosts", "description" : "Enter Maintenance Mode on ESXi Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:27.057Z", "name" : "Get vSphere Cluster MOID", "description" : "Get vSphere Cluster MOID" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:25.467Z", "name" : "Generate Add VxRail Host Internal Model", "description" : "Generate Internal Model For VxRail Host Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:25.403Z", "name" : "Register Current Task", "description" : "Register Current Task" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:25.044Z", "name" : "Add Host Update Logical Inventory Action", "description" : "Update ESXi Host Data in Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:24.260Z", "name" : "NFS Configuration", "description" : "Configure NFS lcm-bundle-repo On VxRail Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:23.218Z", "name" : "Update Host Details", "description" : "Update VxRail Host Details" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:22.936Z", "name" : "Save Esxis To Uber Spec", "description" : "Get VxRail Host from Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:20.578Z", "name" : "AddHostsToInventoryAction", "description" : "Persisting ESXi Host(s) to the inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:20.549Z", "name" : "Acquire Lock Action", "description" : "Acquire Lock for ESXi Host Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-25T11:01:20.498Z", "name" : "Acquire Lock Action", "description" : "Acquire Lock for ESXi Host Addition" } ], "id" : "79dbee35-a229-4753-8b1c-855cb4cd2202" } |
Add Secondary Cluster to WLD
Once a workload domain has been deployed with the primary cluster, a secondary cluster can be added in. First image the nodes and complete first run of the VxRail cluster with the existing Workload Domain vCenter. Then prepare a json file with the details of the cluster being imported. Sample json file is available below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
{ "domainId": "01dc67ce-f59b-4701-86c2-f766469fefe1", "computeSpec": { "clusterSpecs": [ { "name": "VxRail-Virtual-SAN-Secondary-Cluster", "hostSpecs": [ { "ipAddress": "192.168.15.95", "hostName": "rack14-009.vxrail.local", "password": "RACK14pwd!123" }, { "ipAddress": "192.168.15.96", "hostName": "rack14-010.vxrail.local", "password": "RACK14pwd!123" }, { "ipAddress": "192.168.15.97", "hostName": "rack14-011.vxrail.local", "password": "RACK14pwd!123" } ], "datastoreSpec": { "vsanDatastoreSpec": { "licenseKey": "xxxx-xxxx-xxxx-xxxx" } }, "networkSpec": { "vdsSpecs": [ { "name": "VMware HCIA Distributed Switch VxRail-Virtual-SAN-Secondary-Cluster 06126e", "isUsedByNsxt": true, "portGroupSpecs": [ { "name": "Management Network-06126e12-aa3b-41fc-bfa9-791c64defc39", "transportType": "MANAGEMENT" }, { "name": "Virtual SAN-06126e12-aa3b-41fc-bfa9-791c64defc39", "transportType": "VSAN" }, { "name": "vSphere vMotion-06126e12-aa3b-41fc-bfa9-791c64defc39", "transportType": "VMOTION" } ] } ], "nsxClusterSpec": { "nsxTClusterSpec": { "geneveVlanId": 2514 } } }, "vxRailDetails": { "rootCredentials": { "password": "RACK14pwd!123", "username": "root", "credentialType": "SSH" }, "adminCredentials": { "credentialType": "SSH", "username": "mystic", "password": "EvoSddc!2016" } } } ] } } |
Validate the Input Json
Execute the following command to validate the input json file created above.
|
1 2 3 |
curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id> 'https://192.168.15.28/v1/clusters/validations' -d @wld-secondary-cluster.json –insecure {"id":"9177579b-a1f0-4b5c-92b0-523a8a179eee","description":"Validating Cluster Creation Spec","executionStatus":"IN_PROGRESS"} |
Get Validation Status
Execute the following command to get the current validation status of the input json file.
|
1 2 3 4 5 6 7 8 9 10 11 |
curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id> 'https://192.168.15.28/v1/clusters/validations/9177579b-a1f0-4b5c-92b0-523a8a179eee' -d @wld-secondary-cluster.json --insecure | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1363 0 173 100 1190 1650 11349 --:--:-- --:--:-- --:--:-- 11442 { "validationChecks" : [], "id" : "9177579b-a1f0-4b5c-92b0-523a8a179eee", "resultStatus" : "SUCCEEDED", "executionStatus" : "COMPLETED", "description" : "VALIDATE_VXRAIL_CLUSTER_WORKFLOW" } |
Execute Add Secondary Cluster Workflow
|
1 2 3 |
curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id> 'https://192.168.15.28/v1/clusters/' -d @wld-secondary-cluster.json --insecure {"id":"0fbbe91d-921b-45f3-9967-892e1393defd","name":"Add Cluster","status":"IN_PROGRESS","creationTimestamp":"2020-05-26T07:01:38.414Z"} |
Get Add Secondary Cluster Workflow Status
|
1 |
# curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id> 'https://192.168.15.28/v1/tasks/0fbbe91d-921b-45f3-9967-892e1393defd' -d @wld-secondary-cluster.json --insecure | json_pp |
The full output is given below
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 |
{ "resources" : [ { "resourceId" : "01dc67ce-f59b-4701-86c2-f766469fefe1", "type" : "Domain" }, { "resourceId" : "d5c1d009-2ea3-4e70-8812-b99bde92d506", "type" : "Cluster" } ], "resolutionStatus" : "UNRESOLVED", "status" : "Successful", "creationTimestamp" : "2020-05-26T07:01:38.633Z", "name" : "Adding VxRail cluster VxRail-Virtual-SAN-Secondary-Cluster to domain NSXT-1", "isCancellable" : false, "subTasks" : [ { "status" : "SUCCESSFUL", Workflow output below { "resources" : [ { "resourceId" : "01dc67ce-f59b-4701-86c2-f766469fefe1", "type" : "Domain" }, { "resourceId" : "d5c1d009-2ea3-4e70-8812-b99bde92d506", "type" : "Cluster" } ], "resolutionStatus" : "UNRESOLVED", "status" : "Successful", "creationTimestamp" : "2020-05-26T07:01:38.633Z", "name" : "Adding VxRail cluster VxRail-Virtual-SAN-Secondary-Cluster to domain NSXT-1", "isCancellable" : false, "subTasks" : [ { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:10:58.250Z", "name" : "Release Lock for vSphere Cluster Addition", "description" : "Release Lock for vSphere Cluster Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:10:57.877Z", "name" : "Update vSphere Cluster Data in Inventory", "description" : "Update vSphere Cluster Data in Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:10:57.818Z", "name" : "Validate NSX-T Connectivity Action", "description" : "Validate NSX-T component connectivity after NSX-T network configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:10:57.372Z", "name" : "Detach Transport Node Collection Action", "description" : "Detach Transport node collection from cluster" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:53.315Z", "name" : "Create Transport Node Collection Action", "description" : "Create Transport Node Collection" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:52.850Z", "name" : "Create NSX-T Data Center Transport Node Profile Action", "description" : "Create NSX-T Data Center Transport Node Profile" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:52.189Z", "name" : "Create NSX-T Data Center Uplink Profile", "description" : "Create NSX-T Data Center Uplink Profile" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:51.947Z", "name" : "ProxyContractAction", "description" : "Automation Helper Action" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:51.461Z", "name" : "Prepare NSX-T Add Cluster Input Action", "description" : "Generate required configuration to join NSX-T fabric" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:43.920Z", "name" : "VxRailNsxtAddClusterHeader", "description" : "Gather input to add host to NSX-T Fabric" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:43.864Z", "name" : "Validate NSX-T Connectivity Action", "description" : "Validate NSX-T component connectivity after NSX-T network configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:42.817Z", "name" : "VxRailPrepareNsxtInfoFromInventoryAction", "description" : "Validate NSX-T component connectivity" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:42.682Z", "name" : "VxrailGenerateNsxtInputForAddClusterAction", "description" : "Get NSX-T input data" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:42.640Z", "name" : "Release Lock for vSphere Cluster Addition", "description" : "Release Lock for vSphere Cluster Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:42.268Z", "name" : "Update vSphere Cluster Data in Inventory", "description" : "Update vSphere Cluster Data in Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:39.855Z", "name" : "Update Known VxRail Availability Zone Hosts", "description" : "Update VxRail Hosts SSH Key To Known Hosts Configuration" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:06:39.218Z", "name" : "UpdateVxRailManagerSSHKeysToKnownHostsAction", "description" : "Update VxRail Manager SSH Keys to Known Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:30.375Z", "name" : "Update VMCA Cert In VxRail Manager DM", "description" : "Post VxRail VI Domain Creation Update VMCA Certificate In VxRail Manager" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:16.634Z", "name" : "Reconfigure port groups", "description" : "Reconfigure port groups" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:16.225Z", "name" : "Apply vSAN License in vCenter Server", "description" : "Apply vSAN License in vCenter Server" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:16.114Z", "name" : "Apply vSAN DLF in vCenter Server", "description" : "Apply vSAN DLF in vCenter Server" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:14.307Z", "name" : "NFS Configuration", "description" : "Configure NFS lcm-bundle-repo On VxRail Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:13.360Z", "name" : "StartHostSSHServicePluginAction", "description" : "Enable SSH on Hosts" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:11.981Z", "name" : "ValidateVxRailAddClusterInternalModelAction", "description" : "Validate Internal Model for vSphere VxRail Cluster Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:11.085Z", "name" : "GenerateVxRailAddClusterInternalModelAction", "description" : "Generate Internal Model for vSphere VxRail Secondary Cluster Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:10.914Z", "name" : "Register Current Task", "description" : "Register Current Task" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:08.964Z", "name" : "VxRailPopulateAddClusterRuntimeDataAction", "description" : "Populate VxRail Cluster Managed Object Ids for Add VxRail Cluster Runtime" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:08.495Z", "name" : "SaveVxRailManagerPasswordsAction", "description" : "Save VxRail Manager Passwords" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:07.775Z", "name" : "SaveEsxiCredentialsAction", "description" : "Save Esxi Passwords" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:02:07.439Z", "name" : "CommitVxRailClusterObjectAction", "description" : "Commit the Generated Inventory Objects to the Database" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:52.848Z", "name" : "CreateVxRailClusterInventoryAction", "description" : "Create VxRail Secondary Cluster Data in Inventory" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:52.802Z", "name" : "Acquire Lock for vSphere Cluster Addition", "description" : "Acquire Lock for vSphere Cluster Addition" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:49.953Z", "name" : "ValidateVxRailClusterSpecAction", "description" : "Validate VxRail Secondary Cluster Input Specification" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:49.867Z", "name" : "Validate Nsx-T Cluster Input Specification", "description" : "Validate Nsx-T Cluster Input Specification" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:39.566Z", "name" : "ValidateVxRailHostInSpecAction", "description" : "Validate the hosts in the given secondary cluster spec" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:39.441Z", "name" : "GetInputsForValidationFromImportSecondaryClusterSpecAction", "description" : "Collect Input data for the import of secondary cluster" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T07:01:39.101Z", "name" : "ProxyContractAction", "description" : "Automation Helper Action" } ], "id" : "0fbbe91d-921b-45f3-9967-892e1393defd" } |
Remove Host from Cluster
Removing host(s) from VxRail cluster is a two step process – first it needs to be removed from the SDDC Manager and then from the VxRail cluster. The host will need to be re-imaged before it an be added back to a cluster.
Prepare Input Json
Prepare json file with the required host id. Sample json file is given below
|
1 2 3 4 5 6 7 8 9 |
{ "clusterCompactionSpec": { "hosts": [ { "id": "da05e14a-e13a-4178-b797-49a7d20cb49d" } ] } } |
Get Cluster ID
Get the cluster id for the cluster from which the host will be removed.
|
1 |
curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>' 'https://<sddc manager ip>/v1/clusters/bd852932-8938-4e73-a0c4-9e3b4548afa6' --insecure |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
{ "isStretched" : false, "hosts" : [ { "id" : "0ed89841-7f51-4c38-baf3-c9e82c2e45af" }, { "id" : "bc8560d2-4a54-4287-90db-10d0e0a1d2d7" }, { "id" : "59a5e207-5777-4536-8883-422f9b8901b8" }, { "id" : "da05e14a-e13a-4178-b797-49a7d20cb49d" } ], "primaryDatastoreName" : "VxRail-Virtual-SAN-Datastore-9a78e0b9-b1e0-41c9-a78e-8f0851b04839", "name" : "VxRail-Virtual-SAN-Cluster", "primaryDatastoreType" : "VSAN", "id" : "bd852932-8938-4e73-a0c4-9e3b4548afa6" } |
Perform Validation
|
1 |
curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>'https://<sddc manager ip>/v1/clusters/bd852932-8938-4e73-a0c4-9e3b4548afa6/validations' -d @Compact_Cluster.json --insecure | json_pp |
|
1 2 3 4 5 6 7 8 9 10 11 12 |
{ "validationChecks": [ { "resultStatus": "SUCCEEDED", "description": "ClusterCompactionSpecValidation" } ], "id": "95fb93ce-6152-4caf-87ca-eda4976b2819", "resultStatus": "SUCCEEDED", "executionStatus": "COMPLETED", "description": "Validating Cluster Compaction Spec" } |
Execute the Delete Host Workflow
|
1 |
curl -X PATCH -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>'https://<sddc manager ip>/v1/clusters/bd852932-8938-4e73-a0c4-9e3b4548afa6/' -d @Compact_Cluster.json --insecure | json_pp |
|
1 2 3 4 5 6 |
{ "status" : "IN_PROGRESS", "creationTimestamp" : "2020-05-26T07:51:17.580Z", "name" : "Removing host(s) from cluster", "id" : "a8861fca-3c1b-4e66-aeca-c2ba9889c64f" } |
Get Delete Host Workflow Status
|
1 |
# curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>' 'https://192.168.15.28/v1/tasks/a8861fca-3c1b-4e66-aeca-c2ba9889c64f' -d @Compact_Cluster.json --insecure | json_pp |
Full output is given below
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
{ "resources": [ { "resourceId": "01dc67ce-f59b-4701-86c2-f766469fefe1", "type": "Domain" }, { "resourceId": "da05e14a-e13a-4178-b797-49a7d20cb49d", "type": "Esxi" }, { "resourceId": "bd852932-8938-4e73-a0c4-9e3b4548afa6", "type": "Cluster" } ], "resolutionStatus": "UNRESOLVED", "status": "Successful", "creationTimestamp": "2020-05-26T07:51:17.849Z", "name": "Removing host(s) from VxRail cluster VxRail-Virtual-SAN-Cluster", "isCancellable": false, "subTasks": [ { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:18.012Z", "name": "Acquire Lock Contract Action", "description": "Acquire Lock for ESXi Host Removal" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:18.048Z", "name": "Remove VxRail Host Get Inventory", "description": "Get VxRail Host Details From Inventory" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:18.730Z", "name": "Register Task Contract Action", "description": "Register Current Task" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:18.780Z", "name": "Remove Host Get Credentials Action", "description": "Get Credentials for ESXi Host" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:19.437Z", "name": "VcHostDeletionImpactAction", "description": "Validate that vCenter host deletion will not impact NSX-T Edge cluster" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:19.669Z", "name": "Validate Sufficient ESXi Hosts Remain in vSphere Cluster", "description": "Validate Sufficient ESXi Hosts Remain in vSphere Cluster" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:19.783Z", "name": "Update Host Status Action", "description": "Update ESXi Host Status" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:19.857Z", "name": "RemoveNsxtHostHeaderV30", "description": "Gather input for deletion of NSX-T component" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:51:23.088Z", "name": "Delete Transport Nodes", "description": "Deletion of Transport Node configuration from Host" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:29.658Z", "name": "Enter Maintenance Mode on ESXi Hosts", "description": "Enter Maintenance Mode on ESXi Hosts" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:43.726Z", "name": "Remove VxRail Host Update Inventory", "description": "Update VxRail Host Data In Inventory" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:44.253Z", "name": "VxRail Validate Free Host", "description": "Change Host Status To Maintenance If Host Is Not Part Of Any Cluster" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:44.361Z", "name": "VxRail Clean NFS Configuration", "description": "Clean The NFS Configuration" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:45.041Z", "name": "VxRail Remove Host From Known Hosts", "description": "Remove VxRail Host From Known Hosts File" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:45.203Z", "name": "VxRail Delete Host Credentials", "description": "Delete VxRail Host Credentials From Credential Store" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:45.819Z", "name": "Delete VxRail Host From Inventory", "description": "Delete VxRail Host From Inventory" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T07:53:45.862Z", "name": "Release Lock Contract Action", "description": "Release lock for ESXi Host Removal" } ], "id": "a8861fca-3c1b-4e66-aeca-c2ba9889c64f" } |
The workflow can also be monitored from the SDDC Manager GUI. Once the workflow completes, login to the VxRail plugin in the vCenter and remove the host from the cluster.
Password Rotate
Passwords for all the supported components can be rotated via API calls also. Password Management section in VMware Cloud Foundation Administration guide has more details.
Prepare Input Json
Create a json file containing details of components for which the password will be rotated. Sample json file is given below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
{ "operationType": "ROTATE", "elements": [ { "resourceName": "rack-007.vxrail.local", "resourceType": "ESXI", "credentials": [ { "credentialType": "SSH", "username": "root" } ] } ] } |
Execute the Rotate Password Workflow
|
1 2 3 |
# curl -X PATCH -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>' 'https://192.168.15.28/v1/credentials' -d @rotate_password --insecure {"id":"ab4daa28-7689-4ece-8f48-541fc332d847","status":"IN_PROGRESS"} |
Get the Rotate Password Workflow Status
|
1 |
curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>' -d @rotate_password --insecure | json_pp |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
{ "resolutionStatus": "UNRESOLVED", "isCancellable": false, "subTasks": [ { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T09:48:24.656Z", "name": "Password rotate prevalidation", "description": "Prevalidation of password rotate request" }, { "status": "SUCCESSFUL", "creationTimestamp": "2020-05-26T09:48:25.656Z", "name": "rack-007.vxrail.local:SSH", "description": "Password rotate for rack-007.vxrail.local and credential type SSH" } ], "status": "Successful", "creationTimestamp": "2020-05-26T09:48:24.656Z", "name": "Credentials rotate operation", "id": "ab4daa28-7689-4ece-8f48-541fc332d847" } |
The workflow can also be monitored from the SDDC Manager GUI
Update Password
Unlike password rotation, which generates a randomized password, you can provide a specific password for a specific component. More details are available in VMware Cloud Foundation Administration guide.
Prepare Input Json
Create a json file containing details of components for which the password will be rotated. Sample json file is given below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
{ "operationType": "UPDATE", "elements": [ { "resourceName": "rack-007.vxrail.local", "resourceType": "ESXI", "credentials": [ { "credentialType": "SSH", "username": "root", "password": "RACKpwd!123" } ] } ] } |
Execute the Update Password Workflow
|
1 2 3 |
# curl -X PATCH -H 'Content-Type: application/json' -H 'Authorization: Bearer <token id>' 'https://192.168.15.28/v1/credentials' -d @update_password --insecure {"id":"86295164-da53-49e2-af76-bd787ca87483","status":"IN_PROGRESS"} |
Get the Rotate Password Workflow Status
# curl -X GET -H ‘Content-Type: application/json’ -H ‘Authorization: Bearer <token id>’ ‘https://192.168.15.28/v1/tasks/86295164-da53-49e2-af76-bd787ca87483’ -d @update_password –insecure | json_pp
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
{ "resolutionStatus" : "UNRESOLVED", "isCancellable" : false, "subTasks" : [ { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T09:52:43.477Z", "name" : "rack-007.vxrail.local:SSH", "description" : "Password update for rack-007.vxrail.local and credential type SSH" }, { "status" : "SUCCESSFUL", "creationTimestamp" : "2020-05-26T09:52:42.477Z", "name" : "Password update prevalidation", "description" : "Prevalidation of password update request" } ], "status" : "Successful", "creationTimestamp" : "2020-05-26T09:52:42.477Z", "name" : "Credentials update operation", "id" : "86295164-da53-49e2-af76-bd787ca87483" } |
The workflow can also be monitored from the SDDC Manager
Troubleshooting
- If there are any API related errors around input json passed with invalid input, review the log file at /var/log/vmware/vcf/domainmanager/domainmanager.log. Below is a sample error message for invalid input json. Once the input json file has been fixed, execute the same API call to revalidate the json file.
|
1 2 3 4 5 |
2020-05-26T05:28:19.861+0000 ERROR [cdf0d554b3852c23,8702] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7200-exec-6] Spec violation DM_COMPUTESPEC_NULL 2020-05-26T05:28:19.861+0000 ERROR [cdf0d554b3852c23,8702] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7200-exec-6] Spec violation DM_DOMAINID_INVALID_UUID 2020-05-26T05:28:19.861+0000 ERROR [cdf0d554b3852c23,8702] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7200-exec-6] Spec violation DM_DOMAINID_NULL 2020-05-26T05:28:19.862+0000 ERROR [cdf0d554b3852c23,8702] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7200-exec-6] [PULSDS] REST_INVALID_API_INPUT Invalid input com.vmware.evo.sddc.common.core.error.CompositeInvalidInputException: Invalid input |
- If there are any workflow related errors, review the log file at /var/log/vmware/vcf/domainmanager/domainmanager.log to determine the issue. Once the issue has been fixed, retry the task from API following this guidance.
This concludes all the steps required to perform additional worklod domain operations post deployment.
