Enterprise development teams are increasingly under pressure to deliver applications faster, and with greater functionality. Many of them choose to take advantage of convenient storage, queueing, and other such API-accessible, public cloud services to help them in this regard. While Pivotal offers many of the same capabilities natively (e.g., MySQL, GemFire, or RabbitMQ), enabling choice and easy access to public cloud APIs is an ongoing priority for Pivotal Cloud Foundry (PCF).
We are proud to announce that the recently released Service Broker for Amazon Web Services (AWS), is a big step in this direction! As with all service brokers, the Service Broker for AWS implements the well-defined, Cloud Foundry Service Broker API, which enables a generalized means of connecting to services. Based on this framework, enterprises can quickly build a hybrid cloud service marketplace for their developer communities that has access to both on-premise and public cloud services (check out service brokers for Azure and Google Cloud Platforml).
The Service Broker for AWS enables a simplified developer abstraction for creating, consuming and disposing AWS services as well as centralized IT control of security, authorization, governance and AWS billing – all critical enterprise concerns that drive productivity, reduce complexity and help control costs.
In this blog post, we discuss the capabilities and developer experience for the service broker. We end with some code – a fun sample app that exercises the broker and shows it in action! Here’s a Github link to the code.
Service Broker for AWS Overview
The PCF Service Broker for AWS provides developer-friendly abstractions for popular application services that run on AWS. It helps developers to quickly provision and consume these services without needing to 1) coordinate with the central enterprise IT function on AWS security, access and billing or 2) manage the low-level details of setup, configuration, and teardown of AWS resources. Once set up by the IT operator, the service broker automates all these concerns through calls to the AWS API. Developers only need to remember a few PCF commands that they can use to access multiple AWS services.
Today, the broker supports provisioning and managing the following AWS resources from the cf command line:
-
Amazon RDS which offers a choice of database engines for your relational storage needs, including PostgreSQL, MySQL, Aurora, SQL Server, Oracle and MariaDB.
-
Amazon DynamoDB, Amazon’s, low latency, NoSQL service, is ideal for many modern applications. With support for both document and key/value data models, DynamoDB can help you develop web, gaming, IoT and other applications that require support for flexible and rapidly evolving data models.
-
Amazon Simple Storage Service (S3), Amazon’s highly available, highly scalable object storage, ideal for images, bulk storage, analytics, backup/recovery, etc.
-
Amazon Simple Queue Service (SQS), the fast, scalable, persistent and asynchronous messaging system that can form the foundation for modern, microservices-based workloads.
As with all service brokers, the Service Broker for AWS handles command line requests from developers to provision, bind, unbind and delete these resources.
Before developers can use it, it needs to be installed and configured for use with PCF by your Cloud Foundry Ops team. The PCF operator must have an AWS account and appropriate IAM user with relevant roles/privileges set up. The Service Broker for AWS is then installed and configured with the broker’s IAM user credentials so it can invoke the AWS apis. Granular AWS service level preferences (e.g. region, queue message size, etc.) can also be configured. A detailed discussion of the setup procedure can be found here.
The next section provides a high-level overview of the developer experience in working with the Service Broker for AWS.
The Developer Experience
Once the service broker is set up, the developer experience is essentially the same for the Service Broker for AWS as it is for any other PCF service (and consistent no matter where PCF is deployed: on-premise or public cloud). Developers simply log in to PCF via the cf command line interface (CLI) and issue the usual cf commands used to work with PCF services. Since the setup process already configured AWS credentials for the broker to issue AWS API calls (see below), developers do not need to manage API keys directly thereby keeping PCF secure by default.
Developers can get a listing of AWS (and all other available) services available with their PCF deployment through the cf marketplace command.
Using the service and plan names from the cf marketplace listing, service instances can be created and managed through the standard service lifecycle management commands below.
-
cf create-service: Issuing this command results in the instantiation of a service instance and associated (dynamically generated) access credentials that can be bound to application instances (seecf bind-service). Behind the scenes, the service broker issues AWS API calls to provision and configure the underlying AWS resources. As a result, AWS initiates billing for those resources at this time. Multiple service instances can be created for each service (subject to IT Operator specified limits). An example command creating the S3 service instance (named “my-aws-bucket”) using the “standard” plan would be the following,
cf create-service s3 standard my-aws-bucket
An optional “-c” flag can be used to specify additional configuration, e.g. AWS availability zone.
-
cf bind-service: Issuing this command results in “binding” of the service instance to the application. Whereas creating is the act of provisioning and configuring service resources, binding is the act of “making the service accessible” by the application. Although the specific actions taken vary by service, binding usually involves setting resource permissions (e.g. RDS database permissions) and injection of service access information (e.g. access credentials or service specific metadata) as environment variables for each of the running application instances. Binding is a dynamic process – the PCF platform handles supplying this information seamlessly to additional instances as the application scales. The following command binds app “my app” to service instance “my-aws-bucket”:
cf bind-service my-app my-aws-bucket
The service information is made available through the VCAP_SERVICES environment variable (see below for an example).
-
cf unbind-service: As the inverse of thecf bind-servicecommand, this command results in the reversal of actions taken during the bind command, i.e. environment variable settings and resource permissions are removed from application instances. -
cf delete-service: As the inverse of thecreate-servicecommand, thedelete-service
Finally, developers can temporarily gain additional privileges to perform one-off tasks (e.g. purge a queue) using the cf create-service-key which grants predefined privileges for a limited time. This not only minimizes IT overhead, but also enhances developer velocity and simplifies resource management. Additional details on working with each of the services enabled by the AWS broker are available here.
Image Processing Sample App
Let’s look at the above functionality in action through some sample code that processes images! One can imagine the code being the core of an image processing pipeline for a more sophisticated application.
The scenario is simple, and a common one in microservices based architectures: two different processes collaborate via a queue (AWS SQS) and shared storage (AWS S3). In this case, we have two services (both in java): a producer and a consumer. The producer runs locally on your laptop (since it accesses local disk) to generate events that the consumer can process. The consumer runs on PCF and showcases the service broker functionality.
The producer periodically polls a local directory and uploads any image files (files with “jpg”, “jpeg”, “png”, “gif” extensions) that are copied there to the S3 bucket shared with the consumer. It then places a message on the shared SQS queue with the S3 object key. The consumer (running on PCF) monitors the queue. When a new event appears, it removes it from the queue, reads the image from S3 using the supplied object key, generates a border and watermark for the image, and stores the updated image back to the S3 bucket.
Below is a before and after image sample for the image processing being done:
|
|
|
|
Understanding AWS Connectivity
Here’s a snippet of the Consumer’s code that uses environment variables injected during the cf bind command in order to set up its connectivity to AWS:
//Get credentials from the environment
Map<String, String> envVar = System.getenv();
JSONObject obj = new JSONObject(envVar.get(VCAP_SERVICES_ENV_VAR));
JSONObject s3Obj = obj.getJSONArray(AWS_S3_ENV_VAR).getJSONObject(0).getJSONObject(AWS_CREDENTIALS_ENV_VAR);
String bucketName = s3Obj.getString(AWS_BUCKET_ENV_VAR);
String bucketAccessKey = s3Obj.getString(AWS_ACCESS_KEY_ENV_VAR);
String bucketSecretKey = s3Obj.getString(AWS_SECRET_KEY_ENV_VAR);
String bucketRegion = s3Obj.getString(AWS_REGION_ENV_VAR);
…
System.out.println("Initializing AWS Client...");
//Set up credentials and AWS clients
BasicAWSCredentials myS3Creds = new BasicAWSCredentials(bucketAccessKey, bucketSecretKey);
AmazonS3 s3 = new AmazonS3Client(myS3Creds);
s3.setRegion(Region.getRegion(Regions.fromName(bucketRegion)));
The code above reads its environment variables during startup initialization. It looks for the VCAP_SERVICES environment variable and then parses its value (a JSON fragment) to extract the AWS credentials (and other metadata, e.g. region) required to connect to AWS. (The credentials are supplied by the AWS broker dynamically as part of the cf create-service/cf bind-service commands.) It then initializes the AWS S3 client object with the access credentials. A similar process is followed for the AWS SQS client. The rest of the code then uses the initialized AWS clients to invoke AWS apis to perform the image transformations. The code for the producer and consumer can be obtained from Github, here. You will need a running PCF environment to deploy it.
Running the Consumer
To run the consumer, you can simply build it using the supplied pom file (use “mvn package”), and then cf push the resultant fat jar in your /target subdirectory to PCF. Since this is not a web app, you must use the --no-route option and turn the health check off. Also, since the consumer immediately looks for environment variables to connect to AWS, it is important to use –no-start option so you can defer starting until the AWS services are created and bound. Lastly, you will create and bind the AWS bucket and queue that the app relies upon and start the app. Use the following commands to get going:
> cf push Consumer -p <My fat jar> --no-route --no-start > cf set-health-check Consumer none > cf create-service s3 standard kb-bucket > cf create-service sqs standard kb-queue > cf bind-service Consumer kb-bucket > cf-bind-service Consumer kb-queue > cf start Consumer
The cf create-service commands above invoke the Service Broker for AWS to perform the necessary provisioning of AWS resources. The cf bind-service command initializes the VCAP_SERVICES environment variable for the app. A credentials portion of a sample listing of VCAP_SERVICES for a bound application might appear as follows:
"VCAP_SERVICES": {
"s3": [
{
"credentials": {
"access_key_id": "AKIAI6EKD2SSIWYV6TTA",
"bucket": "cf-21aae293-ad9f-4327-b3fe-a151a88f3258",
"region": "us-east-1",
"secret_access_key":"Ghmlzk7DwG4tJ1w5ZXyxFASsMkY/omkmSOreaKkh"
},
...
Running the Producer
Once the Consumer is up and running, it’s time to get the producer going. Compile the producer the same way: mvn package.
The producer will run locally, so you will need to supply the access credentials and resource names that were created for the queue and bucket (remember, the queue and bucket were provisioned on AWS when you invoked cf create-service for the consumer).
For simplicity, the producer’s code assumes the same environment variable and value as the consumer. Inspect your environment variables for the consumer using Apps Manager and just copy the VCAP_SERVICES environment variable to the producer’s environment. The only additional runtime configuration you will need to do will be to add the MONITORED_DIRECTORY environment variable to tell the producer which image directory to poll (every 10 seconds).
Now, just run the producer jar file that maven generated. To see the producer in action, simply copy a few image files to the monitored directory and monitor the producer as it moves them to the S3 bucket and notifies the Consumer. The consumer will also poll its queue every 10 seconds and process the queue messages to transform the uploaded images.
Conclusion
As the sample shows, you didn’t have to log in to AWS, create/manage credentials, set up security policies (and ensure they met central IT criteria), provision and configure buckets and queues, etc. All you had to do was cf create-service and cf bind-service. The rest of your time was spent developing application code! The Service Broker for AWS took care of the details of setting up and managing AWS cloud resources, so you could focus on building apps that leverage powerful AWS capabilities.
