This article is by Hari Sivaraman, Uday Kurkure, and Lan Vu from the Performance Engineering team at VMware.
Performance Comparison of Containerized Machine Learning Applications
Docker containers  are rapidly becoming a popular environment in which to run different applications, including those in machine learning [1, 2, 3]. NVIDIA supports Docker containers with their own Docker engine utility, nvidia-docker , which is specialized to run applications that use NVIDIA GPUs.
The nvidia-docker container for machine learning includes the application and the machine learning framework (for example, TensorFlow ) but, importantly, it does not include the GPU driver or the CUDA toolkit.
Docker containers are hardware agnostic so, when an application uses specialized hardware like an NVIDIA GPU that needs kernel modules and user-level libraries, the container cannot include the required drivers. They live outside the container.