When Pixar creates movies like Toy Story 3, the company has millions of files containing software, movie clips, audio files and many other types of asset that need to be managed, versioned and stored so that they can be retrieved very quickly. Pixar chose to use the Perforce revision control software (P4D) to handle this complex and vital task, and in doing so, they decided to place their Perforce P4D software on VMware vSphere as a standard platform.
I spoke recently with Mike Sundy, Digital Asset Administrator at Pixar and he described his virtualized environment to me. Pixar has been virtualizing their Perforce P4D software for a number of years now with great success. Here are some interesting facts from his description. Pixar grew their Perforce code server instances from 1 to begin with to over 119 instances as of May 2012.
In May 2012, for example, they had:
- About 1000 users
- 83 GB db. have on largest server
- 12 million p4 operations per day
- 37 VMware ESX servers
- 20 million files in Perforce
- 70 Terabytes of p4 storage (including a single 27 TB depot)
Given the industry that the company is in, then the type of data that Pixar stores in their versioning system is not just software files, but also the following:
- Art – both reference and concept art – used as inspirational targets for films
- Department Specific data – such as creative resources data with blessed marketing images
- Company-wide reference libraries, such as animation references, config files, company photos
- Exotics, such as patent data, casting audio, data for live action shots, story gags, theme park concepts, art shows, post production, etc.
This shows that the Perforce software is capable of versioning a lot more than just software developer artifacts, but is in use for versioning almost everything. The key issue the Pixar team faced in this business was being able to quickly scale their Perforce server infrastructure in order to handle these massive quantities of different data that they were being asked to manage. A major factor in achieving success at doing this was their use of virtualization from VMware.
Benefits Derived from Virtualizing Perforce P4D
Pixar staff was able to rapidly spin up new Perforce servers from a remote site in a virtual machine on existing hardware in order to cater for more user load (cloning a VM, installing Perforce, running customization scripts and validating/testing could all be completed within an hour from start.) Valuable datacenter space was conserved when they used virtual machines, which their costs for power, cooling and systems administration. The system is stable and fault tolerant. If a hardware server is due to be brought down, then the running VM is automatically and transparently migrated to another host and the end users never notice.
Perforce Performance on Virtual Machines on vSphere
Several benchmarking exercises were run at Pixar to prove that Perforce P4D could be virtualized safely and provide adequate responses to its users. Results of the tests at that time were that the virtualized system came within 15% of the performance of the physical servers in the worst cases and were close to 95% of native performance in many cases. These tests used the branch submit and browse benchmarks that are very familiar to Perforce administration folks who study performance. Overall, the findings were that the performance was very good and up to the job of serving the company’s needs.
Lessons Learned from Deploying on the Virtual Platform
Making sure that the load stays balanced across multiple nodes is still necessary when you are virtualized (as it is in native implementations). VMware helps by providing the vMotion feature to allow you to move a virtual machine from one ESX host to another. But in order to make sure that the highest loaded P4D servers were distributed across different servers, there was some manual administration required.
Symptoms like a disk space filling up to over 90% causing a slowdown in performance are no different on virtual than they are on native. These required action on behalf of the administrator to get the situation back to normal, whether it is in physical or virtual.
Pixar is continuing to virtualize more and more of its Perforce servers and the vSphere platform is now the installation environment of choice for future use.
Learn more about the Pixar implementation today.