Netflix has been doing quite a bit of work with open source software. In fact, they maintain 34 public repositories on GitHub. Since their development team’s focus on OSS stood out from many other Grails users like Disney, Lyris, Qualcomm, and ValueClick, we wanted to highlight a few of their recent Grails projects.
Lipstick built for a Pig
Certainly, the development team came up with quite a funny name, but the Lipstick project certainly doesn’t look like a joke. In this case, Lipstick does make a Pig look better.
For those working with Apache Hadoop, you are probably also familiar with Pig. If you aren’t familiar, we recently published an article that compares MapReduce, Hive, Pig, and Cascading with the well-known word-count example. Basically, Pig provides a higher level of abstraction for MapReduce programming in a syntax similar to SQL yet can also be extended with functions written in Java, Python, JavaScript, Ruby, or Groovy.
In a nutshell, Lipstick is a Grails app that provides a graphical view of a Pig workflow.
As with most languages, complex applications can become hard to follow—making it difficult for developers to get work done. With Lipstick, it’s easier to maintain, debug, and monitor MapReduce jobs running Pig because we don’t have to sift through so many logs. The service runs on top of all Pig scripts and, in real time, reports status of each step in the workflow as it runs. As seen above, there is a useful user interface that goes well beyond a mouse-over for more info, and the app runs on Tomcat and MySQL.
ICE for Cloud Spending and Usage
ICE is another Grails project by Netflix, and it is built to provide an analytical view of cloud resources from a usage and cost perspective. It allows users to look at trends, organize reports into groups, and make better decisions.
There are three parts to ICE. First, the processor transforms the AWS programmatic billing access data into readable information. Two, a reader renders this data to the user interface. Then, the UI shows the graphs and tables. With ICE, you get visibility into accounts, regions, services, and usage types along with metrics on cost and use.
As you can see from the screen above, there are numerous filters that let you slice and dice interactively—you can see the cost per hour over time in chart views and table views. There are several other views that you can check out on the ICE project homepage.
More Netflix Open Source Projects and a Prize
There are a few other Netflix projects we think are worth checking out like:
- Asgard—This project provides web interface and capabilities for automating deployments and managing clouds stacks. Asgard was the first Grails project at Netflix, and most of the other Netflix projects follow patterns from Asgard.
- Blitz4j—Allows you to convert log4j appenders to use an asynchronous model for scale.
- Genie—Provides a set of REST-ful APIs to run Hadoop, Hive, and Pig jobs as well as manage multiple Hadoop resources.
- Pytheas—A framework for rapidly building dashboards to explore data.
- Zuul—An API gateway processing system using functional Groovy scripts to dynamically maintain a smart routing fabric for requests
The Cloud-Prize contest might be the most interesting one to point out to Grails developers for now. When this contest ends in mid-September of 2013, some talented developer is going to win $10K USD, $5K USD in AWS credits, a trip to Vegas, and a ticket to the Amazon user conference, not to mention some solid notoriety. Basically, you need to build something cool or modify/improve one of their projects. Check out the Cloud-Prize contest page on GitHub for more information.
How to learn more about Grails
- Grails homepage, Twitter account, or This Week in Grails posts
- The Grails Developer blogs
- A slide presentation covering how NetflixOSS pulls all their components together
- Grails plugins for Cloud Foundry, Spring Security, RabbitMQ, Redis, MongoDB, and more
