cloud_foundry products

Wrapping libyaml in go

featured-gopher-libraryWe recently released version 6.0.0 of cf, the command line client for Cloud Foundry. cf was previously written in Ruby, and we have rewritten it in Go. This allowed us to package cf as as a single binary and simplified our deployment strategy enormously.

The YAML problem

Two weeks before our release, we realized that the go YAML library we were using, goyaml, is distributed under the LGPL license. This makes it unusable for cf. We had to quickly find another way to parse YAML in cf.

The definitive YAML implementation is a C library called libyaml. We knew that Go had good support for interfacing with C libraries, so we decided to write our own go bindings to libyaml. This felt a bit risky given that we were releasing in two weeks, so we named our creation ‘gamble’.

Writing the library

Go’s FFI system cgo allows Go programs to use C libraries without adding any ‘wrapper’ code in C. This is a simplified code snippet from gamble:

package gamble

// #include "./yaml.h"
import "C"

func Parse(input string) interface{} {
    var parser C.yaml_parser_t
    C.yaml_parser_initialize(&parser)

    cString := (*C.uchar)(unsafe.Pointer(C.CString(input)))
    cLen := (C.size_t)(len(input))
    C.yaml_parser_set_input_string(&parser, cString, cLen)

    return getNode(&parser, C.YAML_DOCUMENT_END_EVENT)
}

Here, we enable cgo by importing the "C" package. We can can then reference C types (yaml_parser_t, *uchar), functions (yaml_parser_initialize) and constants (YAML_DOCUMENT_END_EVENT) as if they were members of the "C" package. This makes integrating with C libraries feel amazingly effortless.

Using cgo and libyaml, we were able to implement YAML parsing and marshaling in less than three hundred lines (plus tests):

$ wc -l *.go

  12 gamble_suite_test.go
  16 helpers.go
 188 marshal.go
  65 marshal_test.go
  73 parse.go
 172 parse_test.go
 526 total

Building the library

We didn’t want to complicate the build process for applications that use gamble by requiring them to compile libyaml manually. Luckily, when go builds a package, it compiles any C source files in the package’s directory and links them into the binary. This allows us to bundle libyaml source with gamble and avoid any extra build steps. To make this work, we had to include into gamble the macro definitions that would normally be generated by libyaml’s configure script.

Unfortunately, after creating gamble, we learned that cgo did not work with cross-compilation. Though it looks like this has been recently fixed on Go’s master branch, at the time we had to update our CI setup because we could no longer build cf for all of our target platforms from a single machine.

Also, since libyaml uses functions in the C standard library, applications that use gamble depend on libc. With help from the open source community, we have updated our builds to be compatible with most versions of libc that are still in use.

Takeaways

Go’s FFI is a joy to use and will be even more useful when cross-compilation support is officially released. We at Cloud Foundry give our sincere thanks to the go team for their wonderful work.

gamble is distributed under the Apache 2 License. The source is available on github. If you need to parse or serialize YAML from go, please give it a try!

Related Articles