The following is a guest posting from Daniel Hiltgen, one of the senior developers on the ESXi product.
Did you know that you can quickly and easily transfer files to and from ESXi using basic web protocols?
Starting in version 3.5, ESXi supports HTTP based file transfer for files on any mounted datastore. You can try this out with your favorite web browser by navigating to https://$HOSTNAME/ and clicking on the "Browse datastores in this hosts's inventory" link on the right hand side. (Or you can link directly via https://$HOSTNAME/folder?dcPath=ha-datacenter )
Through your web browser, you'll be able to easily navigate the datastores, directories, and download files from the host. You can upload files using any tools that support the HTTP PUT protocol (web editors for example) or you can use command line tools to make the task a little simpler and more scriptable. (Note: directories will be created on the fly if they don't already exist when you upload.) The interface supports the offset header, so you can restart downloads that are interrupted, or logically "chunk" a large file into multiple parts and download them in parallel to improve the download rate. This interface is also supported against vCenter, but for the purpose of this post we'll focus on direct ESXi access.
Lets talk a little about the URL format first before we look at the scripts. Here's an example URL on my system:
https://esx01/folder/NewVM/NewVM-flat.vmdk?dcPath=ha-datacenter&dsName=MyRaid1Datastore
We can break this down as:
- https://esx01 : The hostname
- /folder : Tells the system you're browsing for files on datastores – this is a constant
- /NewVM/NewVM-flat.vmdk : The path relative to the datastore for the file in question
- ?dcPath=ha-datacenter : The datacenter path – always ha-datacenter for hosts, but will vary on vCenter
- &dsName=MyRaid1Datastore : The name of the datastore – "MyRaid1Datastore" in this case
Here's and example shell script that leverages the "curl" tool to support both uploads and downloads of files.
|
1 |
#!/bin/bash |
|
1 |
# Remove the --insecure if you want to enforce the validity of the SSL key on the server |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
CURL_ARGS="--insecure" # Change if you want to use an alternate user # (you'll be prompted for the password each time) USER=root if ! $(which curl > /dev/null); then echo "ERROR: curl not found in your path" >&2 echo "" >&2 echo "You'll need to install curl on your system for this script to work." >&2 exit 1 fi usage() { echo "USAGE: $0 <get|put> <hostname> <ds> <ds path> <local_path> [thread_count]" >&2 echo "" >&2 echo "one of source or target must be a datastore path" >&2 echo "Example: $0 put hostname datastore /file.iso ./file.iso'" >&2 echo "" >&2 echo "If you set the thread_count then this script will" >&2 echo "use that many parallel threads when downloading the file." >&2 echo "Warning: you'll need ~2x the file size in available space locally for this approach" >&2 } urlescape() { TMP=$(echo $1 | sed -e ' { s/%/%25/g s/ /%20/g s/</%3C/g s/>/%3E/g s/#/%23/g s/{/%7B/g s/}/%7D/g s/|/%7C/g s/\/%5C/g s/^/%5E/g s/~/%7E/g s/[/%5B/g s/]/%5D/g s/`/%60/g s/;/%3B/g s|/|%2F|g s/?/%3F/g s/:/%3A/g s/@/%40/g s/=/%3D/g s/&/%26/g s/$/%24/g }' ) echo ${TMP} } if [ $# -lt 5 ] ; then usage exit 1 fi OPERATION=$1 HOSTNAME=$2 DATASTORE=$(urlescape "$3") REMOTE_PATH=$(urlescape "$4") LOCAL_PATH=$5 if [ $# == 6 ] ; then THREADS=$6 else # Default to 4 threads for better performance THREADS=4 fi URL="https://${HOSTNAME}/folder/${REMOTE_PATH}?dcPath=ha-datacenter&dsName=${DATASTORE}" if [ "${OPERATION}" == "get" ]; then if [ ${THREADS} -gt 1 ] ; then echo -n "Enter password for ${USER}@${HOSTNAME}: " stty -echo read PASSWORD stty echo echo "" # Note: This is somewhat insecure as the password will show up on the # command line. Consider switching to use netrc or SSPI # see the curl man page for more details. # # This first curl invocation grabs just the header to get the size LENGTH=$(curl ${CURL_ARGS} -s -u "${USER}:${PASSWORD}" "$URL" -I | awk '/Content-Length:/ { print $2 }'|sed -e "s/r//g") CHUNK=$((LENGTH / THREADS + 1)) START=0 echo "Starting download (${THREADS} parallel threads)..." COUNT=1 while [ ${COUNT} -le ${THREADS} ] ; do END=$((START + CHUNK - 1)) # Progress reporting with multiple threads gets jumbled up, so be silent curl -s ${CURL_ARGS} -u "${USER}:${PASSWORD}" --create-dirs --range ${START}-${END} "$URL" -o "${LOCAL_PATH}.${COUNT}" & COUNT=$((COUNT + 1)) START=$((END + 1)) done wait # Merge the files back together rm -f "${LOCAL_PATH}" touch "${LOCAL_PATH}" COUNT=1 while [ ${COUNT} -le ${THREADS} ] ; do cat "${LOCAL_PATH}.${COUNT}" >> "${LOCAL_PATH}" rm -f "${LOCAL_PATH}.${COUNT}" COUNT=$((COUNT + 1)) done echo "Done" else curl ${CURL_ARGS} -u ${USER} --create-dirs "$URL" -o "${LOCAL_PATH}" fi elif [ "${OPERATION}" == "put" ]; then if [ ! -f ${LOCAL_PATH} ] ; then echo "ERROR: ${LOCAL_PATH} does not exist" >&2 echo "" >&2 usage exit 1 fi curl ${CURL_ARGS} -u ${USER} -T "${LOCAL_PATH}" "$URL" else usage exit 1 fi |
|
1 2 |
<p> </p> |
— Daniel Hiltgen
Related Posts:
