• First steps with Cloud Foundry on Amazon EC2

    • Richard Downer
    • 16 September, 2011
    • Posted in Blog

    Cloud Foundry, if you’ve been hiding from the media blitz, is an effort of VMware to provide a platform-as-a-service for running web applications in the cloud. It supports several web application platforms, most notably Java (Spring) and Ruby. It also provides services such as MySQL and MongoDB. The theory is that you write your web application, declaring the services it depends on, then push it to the Cloud Foundry. It deploys your web application, connects it to the required services, and you can then “scale out” and “scale in” your application simply by issuing a command. One of the things which interested me is that it’s an open source project, CloudFoundry.org, and all the source code is available on GitHub. I wanted to take the open source code, possibly modify it, and start up my own Cloud Foundry cloud. This article explains how I used Amazon EC2 to host a Cloud Foundry.

    Getting started

    My aims are as follows: I want to be able to roll out as many or as few EC2 instances as I want, and have Cloud Foundry run on them. Cloud Foundry has various modules, and a single instance can host one or several different modules, and I want to be able to decide what type of modules run on each instance.

    First, a short disclaimer. At the time of writing, there isn’t a lot of formal documentation for Cloud Foundry that covers what I was trying to acheive, so this is pieced together from postings to the Cloud Foundry forums and some experimentation on my part. This is based on the latest code from GitHub, and cutting-edge code has the risk that it can change – and break – at any time. If you discover any problems here – and hopefully solutions too – please leave a comment and I’ll try and update this article.

    Finally, I’m assuming that you are familiar with using Amazon EC2 and Linux (I won’t describe how to set up SSH private keys with EC2, for instance.) I tested this out with “cutting edge” code from GitHub on 8th September 2011. To be exact, these are the head commit IDs at the point where I tested this process:

    vcap 6a8deac40e9bbc1baad6387f05a5ec121fe5ca13
    java c51b28e90b1c32da89b72f67426011faa4c05902
    services d87e58d69566ddfbaa3a72d70fdd627b1211ab9c
    tests 842d3bff9cd1e8543eb46c19176ab3c1658fda9d

    My plan is to create an AMI which contains the Cloud Foundry installation. Then starting up a new module is simply a case of launching an EC2 instance from my AMI. I’ll use some magic which will allow the instance to determine which Cloud Foundry module type(s) it should be, and start it up on boot.

    Setting up an IP address and domain name

    The next thing you’ll need is a domain name for your cloud, and an IP address to point it at. We’ll start with the IP address. One of the Cloud Foundry modules is the router. This is the entry point into your cloud – incoming requests go to the router, which will then despatch requests to other appropriate instances in the cloud. The router needs a known IP address. An Amazon EC2 Elastic IP address is ideal for this purpose, so use the AWS console to obtain an elastic IP address and make a note of it. Next you need to map the domain name to it. Posts in the Cloud Foundry forum recommend setting up a wildcard DNS entry (like *.example.com), so that every possible hostname in your domain resolves to the router’s IP address. This is because every application that you push to your cloud will have a URL of http://appname.example.com. A wildcard DNS entry isn’t strictly necessary (and my chosen ISP doesn’t allow me to set one up) – if you don’t or can’t use a wildcard, then you will just have to add an ‘A’ record for every application you plan to deploy. If you can’t make a wildcard DNS entry, set up the DNS for your domain to have A records for these hosts pointing at your elastic IP address (naturally, changing example.com for the name of your domain):

    • api.example.com – this, as the name suggests, is the API that the Cloud Foundry tools use to manipulate your cloud.
    • env.example.com – this is the name of the application we’ll use to test the new cloud.

    Security groups

    I haven’t yet worked out a definitive list of the ports that need to be open for the Cloud Foundry modules to talk to each other. During testing, I used a security group configured to open up port 22 (SSH) and port 80 (HTTP) to the Internet, and all the high number ports (1024 through to 65535) to be accessible to (so that other EC2 instances can talk to them.) Obviously, this is not a very secure way of doing things. As I work out what the minimum acceptable set of open ports is, I will update this section. If you have any information about this then please leave a comment below.

    Installing Cloud Foundry

    To start we will need a base AMI. Cloud Foundry is tested against Ubuntu 10.04 “Lucid Lynx” 64-bit and Canonical maintain a searchable list of Ubuntu AMIs. Find a suitable Lucid AMI, and launch it on Amazon EC2. Although Cloud Foundry’s reference platform is 64-bit, I was able to use a 32-bit AMI without any obvious problems, meaning I could use a cheaper ‘small’ instance type (‘micro’ is too small as Cloud Foundry states 1GB of RAM as a minimum requirement.)

    Once it’s started up, log in with SSH as the ubuntu user. We need to bootstrap the Cloud Foundry install. Instructions on how to do this can be found in their README file on GitHub. At the time of writing, the magic invocation is this:
    bash < <(curl -s -k -B https://raw.github.com/cloudfoundry/vcap/master/setup/install)

    This will take a while to run – half an hour or more – so grab a cup of your favourite non-alcoholic beverage while it downloads and installs all of its dependent components. Once it’s finished, you’ll be dropped back to a shell prompt. Log out, and log back in again (you need to do this because the install process changes the .bashrc.)

    Configuring Cloud Foundry

    You now need to tell Cloud Foundry about your router’s IP address and your chosen domain name. Issue this command:
    cd cloudfoundry/vcap
    At this point you will be given a scary warning about an RVM configuration file. If you’re paranoid, view the file – the version in place as I write this is safe and can be accepted with a “yes” answer. Edit each of these files in turn:

    • cloud_controller/config/cloud_controller.yml
    • router/config/router.yml
    • health_manager/config/health_manager.yml
    • dea/config/dea.yml

    For each file, make the following changes:

    • Where you see external_uri: api.vcap.me, change the last part to api.example.com (replacing example.com with your chosen domain name)
    • Where you see local_route:, change the IP address to be the elastic IP address reserved earlier
    • Where you see mbus: nats://localhost:4222/, replace localhost with the elastic IP address reserved earlier

    Testing it out

    At this point we’ll run a test. We should be able to start the services, and then connect to and deploy a basic web application using the correct domain name. Firstly, if you haven’t already done so, use the Amazon EC2 web console to associate the elastic IP address with the running instance. Still on the EC2 instance, issue this command: ./bin/vcap start Go back to the terminal on your local machine. Install vmc if you don’t already have it. You should be able to successfully run the following commands:
    vmc target api.example.com
    vmc info
    vmc register

    Refer to the README file on the vcap GitHub project. At the end of this file is instructions to create and deploy a simple application called env. Follow these instructions, but use --instances 1 instead of 4. Your application should be visible at http://env.example.com/. Once you’ve successfully run it, remove it from your cloud by issuing the command vmc delete env.

    Making it start the right modules at boot

    By default, the Cloud Foundry services are not started at boot. The simplest way to start them is by adding them to the system startup files – /etc/rc.local, but recall that I said earlier we will want to choose which modules run on each instance. The start command just started everything, but that won’t do. We need to be able to make a selection, and have the startup script start only those modules.

    The way I’ve chosen to handle this is with instance user data. EC2 allows each instance to have a set of arbitrary text called user data attached to it. The user data can be set when first launching the instance, and can be modified later when the instance is in the stopped state. We will use the user data to keep a list of the modules we want to start. The startup script is able to read the instance’s own user data and act accordingly. Back on the shell of the EC2 instance, execute the command sudo vi /etc/rc.local. Then insert the following script, after the comments and before the exit line:

    sleep 2s
    runvcap() {
        echo >&2 su -l -c "source .rvm/scripts/rvm && cd cloudfoundry/vcap && ( $* )" $vcap_user
        su -l -c "source .rvm/scripts/rvm && cd cloudfoundry/vcap && ( $* )" $vcap_user
    modules=$(curl -s -k -B
    if [ -z "$modules" ]; then
        runvcap bin/vcap start
        runvcap 'for p in '$modules'; do bin/vcap start $p || exit 255; done'

    Creating the AMI

    That should be everything we need – we are now ready to create our AMI. In the EC2 web console, stop the instance, and wait for it to stop. Then, right-click on the instance, and select Create Image and follow the prompts. Creating the image takes about 5-10 minutes, so just enough time to fetch another non-alcoholic beverage.

    The final test

    Now we can put all the pieces together and run a cloud. First we need to decide how we are going to divide the different modules between the different instances. At the time of writing, the available modules are:

    • router
    • cloud_controller
    • dea
    • health_manager
    • redis
    • mysql
    • mongodb

    The one which I’m particularly interested in is dea – this is the container for my web application. If I tell Cloud Foundry that I want my application to be across 3 different instances, it’s actually 3 dea modules that host the application. So for this example, I’ll run a total of four EC2 instances – my first instance will run every module except dea, and the other three instances will run only the dea module.

    In the EC2 web console, go to the AMIs option in the left menu. Locate the new AMI, right click and Launch Instance. Follow the prompts. When you get to the option which asks for the user data, set it to:

    router cloud_controller health_manager redis mysql mongodb

    Launch your instance, then associate the elastic IP address with it (because it’s running the router module) and wait for the instance to start up. Now, repeat that process, but instead of starting 1 instance, start 3, and set the user data to simply be dea. Launch the instance and wait for it to start up. Go back to the test we did earlier, and repeat it. This time, however, use --instances 3 when pushing the app to the cloud. Visit http://env.example.com/ and you should get a response. Hit refresh several times – you should see that the IP address displayed in the web page varies between the IP address of the three different dea-hosting instances we launched on EC2. Success!


    This forum post, and the posts that it links to, gave me a lot of the technical detail I needed to get started with multiple-node Cloud Foundry configurations.

    Next up figuring out how to add Monterey as a Framework …

Latest Tweets