Today’s guest post is from Iwasaki Yudai, research engineer at NTT Laboratory Software Innovation Center and Du Jun and Zhang Lei from the Cloud Team, Software Engineering Laboratory, Zhejiang University, China (ZJU-SST).
Cloud Foundry is the leading open source PaaS offering with a fast growing ecosystem and strong enterprise demand. One of its advantages which attracts many developers is its consistent model for deploying and running applications across multiple clouds such as AWS, VSphere and OpenStack. The approach provides cloud adopters with more choices and flexibilities. Today, NTT Software Innovation Center and School of Software Technology, Zhejiang University, China (ZJU-SST) are glad to announce that we have successfully developed the BOSH CloudStack CPI, which automates the deployment and management process of Cloud Foundry V2 upon CloudStack.
CloudStack is an open source IaaS which is used by a number of service providers to offer public cloud services, and by many companies to provide an on-premises (private) cloud offering, or as part of a hybrid cloud solution. Many enterprises like NTT and BT want to bring Cloud Foundry to CloudStack in an efficient and flexible way in production environment. Therefore, developing BOSH CloudStack CPI for deploying and operating Cloud Foundry upon CloudStack becomes a logical choice.
Technical Details
Since NTT and ZJU-SST developed BOSH CloudStack CPI independently at the beginning, there are many differences in the implementation. Hence, the first step is to merge code repositories of NTT and ZJU-SST into a new repository. We chose to create a new repository in GitHub cloudfoundry-community in order to encourage more developers to join us.
There are some crucial aspects in the process of refactoring CPI(check out the wiki if you are interested in digging more differences between NTT and ZJU-SST implementations):
- Stemcell Builder
- Create Stemcell
- Basic Zone VS Advanced Zone
- Fog Support
Stemcell Builder
ZJU-SST used standard Ubuntu10.04 ISO file to build stemcells for both MicroBOSH and BOSH. NTT used Ubuntu10.04 of a backported kernel version due to some compatibility problems in their environment. Unfortunately, aufs, which is essential for warden in Cloud Foundry V2, is missing in the backported kernel. So, we decided to try standard Ubuntu12.04 as the base OS of stemcells for both MicroBOSH and BOSH after brainstorming together. We found that, with a minor patch of cf-release, Ubuntu12.04 is compatible with BOSH and Cloud Foundry. The patch only modifies the deployment process of Cloud Foundry, so it does not impact Cloud Foundry itself. In fact, this issue has been solved since cf-release v160 by updating nginx to 1.4.5.
Create Stemcell
When referring to API call create_stemcell in CPI, ZJU-SST used an extra web server as a entrepot to store the stemcell temporarily while uploading, which follows the OpenStack style, but NTT didn’t use an extra web server and took the volume route same as AWS pattern.
Implementation [A]:
- Similar to the OpenStack CPI
- Requires no inception server
- Requires a web server to save qcow2 files and expose them at HTTP for CloudStack
- [Client] –> (SSH upload) –> [WebServer] –> (HTTP) –> [CloudStack]
- The API call can’t receive image files directly, it downloads image files from given URLs. The web server is necessary because CloudStack can not directly receive image data from inception server.
Implementation [B]:
- Same process as that of the AWS CPI
- Requires an inception server to create a bootstrap Micro BOSH stemcell
- Copies (using
dd) a raw stemcell image to a volume attached to the inception server - Creates a template from a snapshot of the volume
Both implementations have cons and pros.
[A] user have to setup a web server somewhere, however users can bootstrap MicroBOSH from the outside of CloudStack.
[B] an inception server is always required, however users don’t have to setup a web server.
After a heated discussion in the open source community, we adopted approach B as the default solution for uploading stemcells to bosh director due to approach A is not user-friendly. Meanwhile, we created a new branch for experiment with approach A.
Basic Zone VS Advanced Zone
Before the collaboration between NTT and ZJU-SST, ZJU-SST worked in CloudStack advanced zone while NTT developed in CloudStack basic zone. ZJU-SST preferred advanced zone because according to our test, basic zone was unable to support Cloud Foundry without applying some “tricks” during the network configuration process. On the other hand, NTT did deploy Cloud Foundry in basic zone by using some “tricks”, for instance, deploying a separated BOSH DNS node and so on. However, we all agreed that it’s inconvenient to work in basic zone especially when we need to redeploy some components such as router. Finally, we reached an agreement to support both network types and add an option to switch between them.
Fog Support
Both NTT and ZJU-SST invoked fog gem to send API requests to CloudStack engine. However, APIs of the official fog gem are not rich enough to supporting BOSH Cloudstack CPI. ZJU-SST built a local fog gem which added the missing APIs while NTT made a fog patch with missing APIs in CPI project to work around for the moment. We already have sent a PR to fog and are waiting for it to be merged.
When refactoring work finished, we started a month-long heavy test. Once a bug was found, the bug finder would open an issue and describe detailed informations about it. Then all of the developers will receive the message about this bug via mail list. Any commit to the code repository would be submitted in the form of pull request and the repository owners would review the set of changes, discuss potential modifications, and even push follow-up commits if necessary. Only if the PR passed CI and BAT can it be merged. Simply put, we followed the workflow of other Cloud Foundry repositories in GitHub such as cf-release and bosh. In this way, we controlled the history of the new repository and prevented potentially dangerous code from being added in.
Current Status
- Finished development and test work.
- Support both basic zone and advanced zone for CloudStack.
- Tested on CloudStack
4.0.0and4.2.0with KVM hypervisor. - Successfully deployed Cloudfoundry
V2and had applications running. - Support both
Ubuntu10.04andUbuntu12.04stemcells. - Support
Ubuntu14.04stemcell is in the TODO list. - Open sourced and maintained in GitHub.
How to Deploy Cloud Foundry on CloudStack
Steps
- Create Inception Server
- Bootstrap Micro BOSH
- Deploy Cloud Foundry
- Setup Full BOSH (Optional)
Create Inception Server
You need a VM on the CloudStack domain where you install a BOSH instance using this CPI. This VM is so-called “inception” server. You will install BOSH CLI and BOSH Deployer gems on this server and run all operations with them.
Why do I need an inception server?
The CloudStack CPI creates stemcells, which are VM templates, by copying pre-composed disk images to data volumes which automatically attached by BOSH Deployer. This procedure is same as that of the AWS CPI and requires that the VM where BOSH Deployer works is running on the same domain where you want to deploy your BOSH instance.
Create Security Groups or Firewall Rules
You also need to create one or more security groups or firewall rules for VMs created by your BOSH instance. We recommend that you create a security group or firewall rule which opens all the TCP and UDP ports for testing. When in production environment, we strongly suggest setting more tight security group or firewall rules.
Boot an Ubuntu Server
We recommend Ubuntu 12.04 64bit or later for your inception server. For those who use Ubuntu 12.10 or later we recommend to select Ubuntu 10.04 or later for the OS type while creating instance from ISO file or registering VM templates.
More Steps
You may find more detailed guide step by step concerning on deploying Cloud Foundry on CloudStack here. In fact, the remaining steps are very straightforward and similar with other deployment methods such as AWS, Vsphere and OpenStack.
Why NTT and ZJU-SST?
This work is powered by NTT and ZJU-SST, we have been working together since last November. Thanks to NTT team member Iwasaki Yudai and ZJU-SST team members Du Jun and Zhang Lei, the main contributors of this project, who had devoted much of their energies to fixing issues and rising to challenges on the CPI work. In addition, we would like to appreciate the selfless help received from Pivotal Cloud team and Nic Williams.
ZJU-SST is the biggest software engineering team of Zhejiang University as well as the leading Cloud Computing research institute in China. ZJU-SST started R&D work on Cloud Foundry and CloudStack about 3 years ago, and more recently launched a comprehensive PaaS platform based on Cloud Foundry V1 serving City Management Department of Hangzhou China. ZJU-SST released BOSH CloudStack CPI for Cloud Foundry V1 last May, and introduced the CPI work at PlatformCF 2013 in Santa Clara, California last September.
NTT, the world’s leading telecom company, has been active in fostering the Cloud Foundry developer and user community in Japan. NTT has been contributing to Cloud Foundry for the last two years and sharing its projects such as Memcached Service, Nise BOSH, a lightweight BOSH emulator and BOSH Autoscaler, with the community. NTT Communications, a subsidiary of NTT Group, has been running a public commercial PaaS Cloudn with Cloud Foundry since March 2013, and a video about their efforts on Cloud Foundry in building a commercial service with it is available at the Cloud Foundry Summit 2013 site.
The decision to work together was motivated in part because ZJU-SST intended to upgrade their previously released CPI to support Cloud Foundry V2 and NTT wanted to improve their independently developed BOSH CloudStack CPI project so that can be compatible with CloudStack advanced zone.
Xiaohu Yang, Executive Vice Dean of School of Software Technology, Zhejiang University, thought highly of this international collaboration. “It will be a win-win cooperation. Open source projects such as Cloud Foundry can serve as an international platform for education and researching”.
Facts and Lessons Learned
This is a successful international collaboration which benefits both NTT and ZJU-SST a lot. With the help from NTT, ZJU-SST is able to release an upgraded CPI which supports Cloud Foundry V2 and CloudStack 4.0/4.2. NTT appreciates ZJU-SST for their huge effort in building a CPI runnable on various CloudStack environments. The most precious assets we get from this cooperation maybe the experience in how to perform international cooperation effectively and how to reach out to the community if help is needed.
Join Us
Questions about the BOSH CloudStack CPI can be directed to BOSH Google Groups. Of course, please open an issue or send a pull request if you want to improve this project.
