Duncan will be delivering two sessions at Cloud Connect, on the critical nature of policy, in addition to code, for the successful management of a cloud application, and a Cloud Solutions Theatre session on the current usage of PaaS in business and the opportunities this presents.

Cloudomatics, or Why Policy Not Code Should Be King
Monday, February 13, 2012, 1:35 PM-1:55 PM. Location: Grand Ballroom E.

Pawn to King 4 (or e4 in algebraic chess notation) is a classic opening move, but in a decent game of chess it is just the start of in an intricate interplay between skilled opponents. So too with cloud. Deploying an application is just the first move in a sequence intended to ensure it continues to deliver the quality of service required while operating within whatever particular jurisdictional or governance regime required. What is critical is not the code but the policies that are provided or referenced to ensure its ongoing management within its operational envelope.

Doing Business in the Cloud
Wednesday, February 15, 2012, 3:35 PM-3:55 PM. Location: Cloud Solutions Theater.

This session will be of interest to you if you are considering standing up a new PaaS, if you are keen to exploit the increased usage and maturity of PaaS solutions or if you want to find out more about the best PaaS management tools currently available.

NEW YORK and EDINBURGH, January 18, 2012 – Cloudsoft Corporation, a software innovator in enterprise platforms for cloud, today announced they have appointed Lawrence Rosen to their strategic advisory board. Rosen joins the board to help shape Cloudsoft‘s open source software strategy.

Rosen has a distinguished track record furthering the adoption and development of open source software and standards. He is a prominent attorney and founding partner of Rosenlaw and Einschlag, specializing in open source and intellectual property. He has served as general counsel and secretary of the Open Source Initiative (OSI), a non-profit corporation dedicated to managing and promoting the Open Source Definition for the good of the community; as board member and legal counsel for the Apache Software Foundation (apache.org); and as board member of the Open Web Foundation (openwebfoundation.org).

Rosen has also taught at Stanford University, authored Open Source Licensing Software Freedom and Intellectual Property Law, and is advisor to a number of open source companies, including Black Duck Software and Jaspersoft Corporation.

Rosen’s timely appointment complements Cloudsoft’s increasing involvement with the open source community centered on jclouds, the open source multi-cloud library aimed at Java, Clojure and other JVM language developers.

“Larry has made a significant contribution to the widespread adoption of open source software by enterprises,” said Duncan Johnston-Watt, founder and CEO, Cloudsoft. “His experience will be invaluable as we work ever closer with the open source community, furthering the adoption of cloud, open standards and open source strategies.”

• 8th to 10th February:
  Citerus are running a three day, hands-on workshop in portable cloud development, lead by Adrian Cole.
• 15th February (Afternoon):
  As part of Jfokus, Andrew Philips will be explaining how to develop portable PaaS applications, and sharing experience of having done so.
• 15th February (Early Evening):
  Meetup for any local jclouds users/developers (and those about to catch flights away from Jfokus) with Adrian and Andrew.

Citerus Workshop

Citerus’ three-day workshop will be led by Adrian Cole (jclouds founder and CTO-jclouds, Cloudsoft. The syllabus is ideal for any Java developer looking to get started consuming cloud resources, as well as those with particular cloud compute or cloud storage use cases.

This class is a must for developers who wish to understand how to portably use services like Amazon S3 and EC2.

Space is still available. Details on Citerus.se (Swedish | English) (Workshop delivered in English.)

Jfokus

Jfokus 2012 runs from 13th until 15th February at the Stockholm Waterfront Congress Centre. Andrew Philips (jclouds Developer) will be speaking on day three of the conference on “Developing Portable PaaS Applications” (Wednesday 15th, 14:00, Room C3)

The jclouds community have written portable Java apps that work across many PaaS offerings* and in this session Andrew will share experience of developing and managing those applications.

In addition to comparing PaaS vs IaaS, public vs private, and various PaaS APIs, Andrew’s talk will look at the multi-PaaS context of live-from-IDE development, continuous integration and deployment, monitoring and management across multiple PaaS.

*(App Engine, CouldBees RUN@cloud, Amazon Elastic Beanstalk, OpenShift, Cloud Foundry and Heroku, to name a few.)

Developer Meetup

As the Closing Keynote of Jfokus ends, grab your things and get over to the jclouds Stockholm Meetup. This will be a relaxed opportunity to talk jclouds, cloud, PaaS-IaaS and all things Cool Code, with lightning talks on jclouds in action. Adrian, Andrew and the Citerus team will be there to talk about jclouds 1.3 and support Camel, Whirr, Pallet, and more.

Watch out Stockholm, jclouds is coming!

via: [ jclouds Blog, Citerus ]

The release comes just in time for jclouds’ Stockholm February, including presentations at JFokus and Europe’s first jclouds training, hosted by Citerus.

Red Carpet features in v1.3 include:

• Support for CloudStack 2.2.13+
• Support for NineFold Compute
• Support for vCloud Director 1.5 endpoints
• Support for OpenStack Nova, via jclouds’s Eucalyptus support

Under the covers there have been a number of improvements, including:

• Enhancements for Openstack Diablo Nova API
• Enhancements to Rackspace Cloud Load Balancers
• Updated to Guava 11.0,  Gson 2.1, and sshj 0.7.0
• Solid State Drive (SSD) support for the Cloudsigma Zurich
• Ability to automatically assign public IP addresses (useful for OpenStack Nova).
• Support for adding/removing nodes from Cloud Load Balancers .
• Support for AWS Cluster Compute Eight Extra Large Instances (cc2.8xlarge)

Full details about the release are included in the jclouds 1.3 release notes and usage examples can be found on GitHub.

Could jclouds be the abstraction layer that completes Cloudsoft?
The pioneering middleware company specializes in cloud-enabling large-scale, distributed, transactional apps. Its latest mission is to support the jcloud library.

Download the report. [ PDF ]

[ via: 451 Research ]

Beyond Big Data – Addressing the Challenge of Big Applications

The challenge of handling big data has spawned a wealth of new data stores and file systems, addressing the challenges of scale and distribution, wide-area and availability, consistency and latency trade-offs. If the data isn’t accessed too frequently, and if the infrastructure doesn’t have to grow, these technologies can mean “job done”.

More often, however, solving the big data challenge is only part of the solution:  most of the time, if you have big data, you’ll have one or more a “big apps”, and sooner or later (better sooner than later) you’ll have to address three more questions:

The first question revolves around how you will compute and deliver this data; the second causes one to question how to operate in or across different infrastructure environments and the third invokes questions about monitoring system behavior and topology changes.

Computing Big Data

The first question follows from the fact that every read and write against data involves compute.  In some systems, this computation adds up and quickly becomes the bottleneck; so even if the storage can scale to petabytes, the number of concurrent requests might max out at an unacceptably low level.

The solution is to design for processing scalability, and broadly speaking there are two popular approaches.  In grid computing, “jobs” or “tasks” are given to “worker” nodes, and scalability becomes a question of scaling the number of such worker nodes.  Hadoop is a powerful extension of this approach where each job is decomposed into multiple tasks each of which runs as near to its target data segment as possible.

The actor model is a more general approach to scalable processing which has lately seen a resurgence in popularity, in part because it can address grid-style compute as well as more sophisticated cases where a unique serialization point is required or shared memory / datastore lookup is too expensive.  In the actor model, application code is decomposed into individual actors, and at runtime messages are passed to relevant actors or chain of actors.  In some ways this is similar to SOA, although often on a much more finely-grained scale.  Actor model systems, such as Erlang, Smalltalk, and Akka, are frequently championed for simplifying the design of applications with good scalability and robustness characteristics — particularly in the face of concurrency and distribution — because of how they force code to be structured.

When working with big data, the important aspect is that actors can be situated near the most relevant data; and with some frameworks, these actors can be moved at will, locally or wide area, with negligible overhead.  This capability allows processing to be optimized in real time for scale, latency, or even cost of compute resource.

At a high level, the thrust of this question is to ensure that, whatever big data solution is being used, the processing fabric is also sufficiently elastic and resilient to handle the corresponding compute load.  Some data systems include some of the above approaches (such as Hadoop), but a crucial part of the architect’s job is to make sure that the compute strategy suits not just the data but also the consumers.

Operating in Hybrid and Multi-Cloud Environments

The second question is a pragmatic one, recognizing that deploying and operating at scale often means using heterogeneous resources, from local hardware or virtualization through to off-site private and public clouds.  In order to use these resources, the application — at least for its deployment — has to interact with the providers and navigate the various models of compute, storage, and networking.  When designing for resilience, or operating at scale, or optimizing for performance or cost, some of the subtleties in these implementations can be quite dramatic.

One way of tackling this is to standardize at the virtualization level across an organization, including external suppliers.  CloudStack, OpenStack, and vCloud are some of the leading choices in this area.  When working in such an environment, the application merely need be designed around that provider’s model and built against their API.

This solution brings its own difficulties, however:  these virtual infrastructure platforms are still quite rapidly evolving, and the API’s change from release to release.  Worse, these changes are not always backwards compatible.  It also requires a heavyweight dev/test environment, when developers might prefer a lightweight local implementation such as VirtualBox to test against.

A complementary approach which can deliver the best of both worlds is to use a cloud abstraction API, such as jclouds (Java), libcloud (Python), or Deltacloud (REST).  These projects present a uniform concept model and API for use in the application, with implementations that allow the code to work against a wide range of cloud providers and platforms.  This means the choice of infrastructure becomes a runtime choice, not a design-time choice; and writing big apps for portability or spanning across multiple clouds becomes natural and consistent.  Furthermore, in many cases, the provider-specific implementations include additional robustness and performance consideration which makes the application developer’s life even better.

The rise of Platform-as-a-Service is another reason to consider an abstraction API.  It is true that writing for a PaaS can greatly accelerate development and insulate an application from the subtleties of different physical and virtual infrastructure layers (and in many cases this they do this by using jclouds or libcloud).  However it can cause lock-in at a different level:  an application designed for a specific PaaS can be very tricky to port to a different PaaS.  Cloud abstraction APIs can prevent against this by making applications portable from the outset, whether targetting infrastructure or PaaS.  Some PaaS entities, such as load-balancers and blobstores, are already available in both jclouds and libcloud, and because these projects are open source, new abstractions can and almost certainly will be emerging.

Effecting Topology Change

One of the most exciting facets of cloud is the ability to have new IT infrastructure “on tap”.  Unfortunately, however, simply having this tap doesn’t mean that applications will automatically benefit.  Designing apps so that they can take advantage of flexible infrastructure — and ultimately communicate how much infrastructure they want — is one of the biggest unsung challenges of cloud.

The naïve answer is not to think outside the box, but to make the box bigger (whether a VM or VLAN or virtual disk).  But with Big Data and Big Apps, this scaling up hits its limit far too early:  the only viable option for scaling is to scale out, that is to get more boxes and to be able to use them.

Recognizing the need for more capacity is not difficult:  the application will respond slowly or not at all, and will report errors.  What is substantially more difficult is to anticipate this need, and more difficult still, to get new capacity online and ready in time.

NoSQL data fabrics, such as Gemfire and Terracotta, can simplify part of this issue, making it easier for applications to incorporate new compute instances, but they tend not to pass judgment on when or how to request these instances (or give them back).  Equally, PaaS offerings can be a good answer where an application’s shape fits a common pattern, such as at the presentation tier.  However in the realm of Big Apps, with large data volumes and multiple locations, the shape and the constraints tend to be unique and the problem remains with the application designers.

The answer in practice is almost always some combination of a CMDB, one or more monitoring frameworks, and scripts to detect and resolve pre-defined situations.  CFEngine, Puppet, Chef and Whirr are noteworthy emerging players tackling various parts of this motley strategy, but even with these tools, writing good scalability and management logic for an application is no small undertaking, even when the management policies are relatively straightforward.

That said, it is an unavoidable part of writing a modern application.  The following collection of suggested best practices is the most which can be said:

• Design so that anything can be changed, with as little disruption as possible
• Be consistent across how the application is initially rolled-out and how changes are subsequently effected
• Decentralize the monitoring and management, pushing logic out to be near the relevant application components
• Consider “infrastructure-as-code”, so that the deployment topology can be tracked and, ideally, replayable
• Consider “policy-as-code”, as the logic which drives an application’s elasticity, fault tolerance, and run-time optimization is an important part of an application (especially with big applications)
• Treat the above “code” like any other code, with version control and testing
• Keep these pieces small and hierarchical, modular and substitutable
• Watch out for new developments in this space, as the current level of difficulty is not sustainable!

Conclusion

In this article we’ve looked at the key design questions facing Big Apps, the flip side of big data: put simply, how do you ensure that having made the right choice at the data tier, the system doesn’t fall down at the processing tier or the infrastructure layer? The triangle diagram summarizes one way of addressing this, focusing on a virtuous cycle of provisioning,  middleware and management. Getting this right delivers a robust, powerful runtime environment, where Big Apps can get the most out of big data.

About the Author

Alex Heneveld, CTO and co-founder of Cloudsoft, brings twenty years experience designing software solutions in the enterprise, start-up, and academic sectors. Most recently Alex was with Enigmatec Corporation where he led the development of what is now the Monterey® Middleware Platform™. Previous to that, he founded PocketWatch Systems, commercialising results from his doctoral research.

Alex holds a PhD (Informatics) and an MSc (Cognitive Science) from the University of Edinburgh and an AB (Mathematics) from Princeton University. Alex was both a USA Today Academic All-Star and a Marshall Scholar.

What is Apache Whirr?

Whirr is a library and a command line tool that can be used to run distributed services in the cloud. It simplifies the deployment of distributed systems on cloud infrastructure, allowing you to launch and tear-down complex cloud cluster environments with a single command.

Supported services currently include most of the components of the Apache Hadoop stack, elasticsearch, Apache Cassandra, Voldemort and Hama. Services can be deployed to Amazon EC2 and to Rackspace Cloud.

What’s New?

This release is packed with new features, service improvements and bug fixes. Particular highlights include:

Puppet & Chef

Whirr supports both Puppet & Chef as configuration management systems. Among other things, this gives you the option to easily deploy a huge range of pre-written services created by other developers. [WHIRR-49, WHIRR-385]

Apache Mahout as a Service

It is clear that a lot of people are using Apache Hadoop for running machine learning algorithms. Whirr allows you to deploy both Hadoop and the Mahout client in minutes. [WHIRR-384]

Ganglia as a Service

Ganglia is a scalable distributed monitoring system for high-performance computing systems. Whirr can automatically configure monitoring, making it easy to keep track of what your cluster is doing. [WHIRR-258]

A complete list of changes is available in the release notes.

Acknowledgments

Thanks go to David Alves, Paul Baclace, Joris Bontje, Adrian Cole, Bruno Dumon, Alex Heneveld, Patrick Hunt, Soren Macbeth, Andrei Savu, Frank Scholten, Karel Vervaeke and Tom White for making this release happen.

Whirr at Cloudsoft

Cloudsoft’s mission is to bring business to the cloud. We are supporting development of Whirr because we see increasing use of complex cloud and map-reduce solutions to big-data problems, and we recognise the need for excellent tools which simplify the creation of sophisticated computing systems from flexible cloud resources.

Andrei Savu, Software Engineer at Cloudsoft, has been actively involved with Whirr since it was in the Incubator at v0.3.0. He said: “This is a great release where we have been able to ship so many new features and improvements. Thanks are due to the awesome team that made this possible. Now work will start on 0.8.0 with exciting features like resizing existing clusters.”

Getting Involved

Whirr is open source and free to download. Contributors, users and visitors are always welcome on the Whirr email list. If you are using Whirr and need to tweak it to match your needs then the “How to Contribute” page details the build-test-commit-review process.

Monterey V4 embraces the Actor Model of concurrent processing, while delivering the scaling, distribution and mobility features available in Monterey V3.

This release includes:

• Actors can migrate between venues, while guaranteeing Monterey’s in-order, exactly-once message delivery and minimal pause time in the actor’s processing.
• Monterey networks can use multiple brokers (e.g. one per location). Brokers can be added/removed on-the-fly.
• Venues can be dynamically re-wired to use different brokers.
• Concurrent transition control: attempts to execute transitions (e.g. migrations) concurrently will be handled correctly
• Qpid broker is now supported in addition to ActiveMQ.
• Venues/actors publish metrics.
• Two migration modes are supported.

This release is available to download from Cloudsoft’s Maven repository. Further details are available on the Download Monterey page.

The release notes detail all the new features, upgrade instructions and known issues.

In Version 4.0.0 Milestone 3, we look forward to:

• Embeddable venues, for running inside AS7, tc Server, POJO JVM apps, etc.
• Improved scalability and performance.
• Improved robustness.
• Health-checking for venues, to detect failures.

CloudBeat is a cloud computing conference from VentureBeat that follows a unique user-centric, case study approach, and acts as fantastic melting pot where users, entrepreneurs and investors can meet and discuss ideas.

This year’s conference will be held in the Hotel Sofitel, Redwood City, California from the 30th of November to the 1st of December. We look forward to seeing you there.

PAX, the Project Automation Experience, is a developers’ conference covering the latest automation tools and techniques for the Java platform. This year’s conference runs from Tuesday 29th November to Friday 2nd December at the Westin, Fort Lauderdale, Florida, USA.

• 1/2 Day Workshop: Hands on Cloud Storage
  Wednesday 30 Nov, 08:30 – 12:00, Atlantic V.
  You may have heard about cloud storage offerings such as Amazon S3, OpenStack or Microsoft Azure. While conceptually similar, these offerings have different APIs and behavior that place the “write once (run|test) anywhere” mantra at risk.The jclouds open source Java and Clojure library aims to eliminate cloud vendor lock-in, exposing easy to use, portable, and powerful APIs. Bring your laptop, armed with latest revs of Eclipse, Git and Maven, and we’ll walk through getting you setup to hack jclouds Java or Clojure BlobStore applications in a collaborative fashion. 

  During this workshop, you’ll discover the value and key gotchas of cloud storage providers first hand. By the end of this session, you’ll be writing testable code that creates and manages containers and blobs, and understand how cloud storage can fit into your architecture.

• Portable Cloud Storage with jclouds
  Thursday – 1 Dec, 13:30 – 15:00, Himarshee.
  Key/value stores are the most common storage offerings in the cloud today. While conceptually similar, BlobStores present different programming models and consistency models that must be considered in application design. After this session, you’ll understand these differences, and know how to use jclouds to avoid cloud lock-in and increase testability without restricting access to cloud-specific features. 

  We’ll start by comparing public cloud services like Amazon S3 or Microsoft Azure and private cloud software like OpenStack Swift or Eucalyptus Walrus from a feature and code/API level. We’ll then review Java and Clojure calls to the open source jclouds BlobStore API, abstracting away these differences. We’ll finish with a review of integration patterns in production today you can consider while designing your cloud architecture.

• DIY NoSQL: Spinning up services with Whirr
  Thursday – 1 Dec, 19:30 – 21:00, Himarshee.
  Whether it’s HBase, Cassandra or one of the many others, you’ve probably already heard about NoSQL. Perhaps you have a continuous test flow dependency, yet are concerned about learning curve or infrastructure required for the NoSQL store you need.During this talk, you’ll see how you can transform infrastructure clouds such as Amazon EC2 or GoGrid into your data cluster using Apache Whirr. 

  We’ll start by reviewing cloud provisioning and how Whirr adds service management to the underlying jclouds ComputeService API. You’ll then learn whirr syntax and how to use it from the command line or via a Java API. Finally, we’ll discuss patterns for workflow integration. When you leave, you’ll know how to setup and use NoSQL stores with a lot less work.

On Meetup.com:
Hands on Cloud Storage Workshop @ PAX Fort Lauderdale
Portable Cloud Storage with jclouds @ PAX Fort Lauderdale
DIY NoSQL Services @ PAX 2011

EDIT: Updated to reflect schedule changes at PAX.