Subscribe to this blog

About Author

Written by staff at Sirius Corporation, the Open Source services group, this blog seeks to dispel any FUD around the use of Open Source software in the Enterprise and provide perspectives on business, economics, politics, philosophy and the environment.

Contact Author

Email Mark

Twitter Profile

Linked-in Profile

Eucalyptus: the unsung hero of Open Source?

Published 10:13, 02 March 09

Tags: applications, open source, windows

Subscribe to this blog

Eucalyptus is an open-source infrastructure for the implementation of cloud computing on computer clusters. Its name is an acronym for "Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems".

The current interface is compatible with Amazon's EC2 cloud computing interface. Tom Callway speaks to Rich Wolski, the project's director, about how Eucalyptus can be leveraged by enterprises and where it sits along side proprietary alternatives like Windows Azure.

1) Can you tell us what led you and your team to develop Eucalyptus?

Yes, certainly. Eucalyptus was developed to support research we have been doing with the VGrADS project. The goal of VGrADS has been to develop a programming and execution environment that makes large-scale grid programs easier and more efficient to write.

As part of this effort, we have been working with the LEAD project and its large-scale weather forecasting applications. We had been having good success in developing software middleware that enabled LEAD workflows to self-schedule across heavily used batch systems on the NSF TeraGrid. We decided that we also wanted to include cloud resources in the resource mix available to LEAD. To do that we needed a development and execution environment for ourselves and the various partner institutions on the project. The idea we had was to run LEAD on the VGrADS execution system (called vgES), and then to alter vgES so that it could include resources from multiple clouds.

The only cloud we could use from the commercial sector, at the time we were architecting the system, was Amazon's AWS. Thus we decided to build an AWS work-alike for the Universities participating in VGrADS, do the vgES port to AWS, and then to run the application on the TeraGrid, Amazon AWS, and the various University machines while they were emulating the AWS cloud. Eucalyptus was the infrastructure we built to support the effort at the University sites.

Because we couldn't change AWS and we didn't have the resources to specialise vgES for more than one cloud API, the quality of the AWS mimicry had to be pretty high. Our goal was to have vgES run in exactly the same manner on AWS and the other Eucalyptus clouds without modification or special purpose code for either.

BTW, we demonstrated the effort at SC08 in November on the TeraGrid, Amazon AWS, and 4 partner Universities/research sites (UCSB, Rice University, RENCI, and University of Tennessee). It was really a great moment to see it all work.

2) Does Eucalyptus sit within or outside UCSB Computing Department's academic curriculum?

Both. UCSB is a research University which means we have a mandate (as UCSB professors) to blend the fruits of our research efforts with our curricular activities. Thus when I teach a graduate operating systems class or a class on scalable systems, Eucalyptus plays a central role. In addition, I have plans for using it as the basis for a graduate class on cloud computing. Development of a new class is quite a time consuming endeavor, however, so it might be a little while before I feel comfortable testing it out on the students.

However we also have a pedagogical obligation to ensure that what we teach is "worth" the students' time. Many excellent research ideas forward science, but are quickly superseded by better ideas that build upon them, or are in an area that will take several years to mature. The students' time is better spent on concepts and systems that have proven longevity and impact. Several aspects of the research we are doing with Eucalyptus have this "not ready for teaching" quality and hence remain outside the curriculum but inside the research lab (for now).

3) In simple terms, how does Eucalyptus work?

It is a set of communicating web services. Users make requests to a front-end web service component called the "cloud controller." If the requests are for storage, they are immediately shunted to Walrus (the front end for the storage subsystem). Otherwise they are translated into a Eucalyptus-internal representation and then handled by web services running at the cluster level, and on the individual compute nodes. On the storage side, the architecture is similar. External requests are translated into an internal representation that is then forwarded to storage controllers running at the cluster level.

The key notion, though, is that the protocols are implemented using unmodified, standard web service technologies. Thus the installation challenge is to get so many cooperating web services up and communicating in a way that is secure. We spend a lot of time on configuration management as a result.

4) Why did you decide to open source the Eucalyptus codebase?

We originally designed Eucalyptus as a tool for exploration of cloud principles. We believe that cloud computing is an important new paradigm, and that the way to accelerate the benefits in can bring to society is to stimulate experimentation and development. In our own professional lives, we look to open source tools and infrastructure to form the backbone of the systems we build and use as the basis for our research innovations. We saw an opportunity to provide the same basis for innovation to others by releasing the code as open source.

5) Why did you choose Linux as the preferred OS?

Two reasons, really. First, the science community has a significant investment in Linux, particularly for HPC. In VGrADS, we already had a significant Linux code base so we certainly needed Linux OS support. Secondly, at the time we started, Amazon AWS only supported Linux. They added Windows shortly after we had begun the effort but since we wanted to access AWS through the VGrADS code base, it made sense for us to stick with Linux.

6) What is Eucalyptus' functional relationship with Amazon EC2?

It is strictly at the API level. We are very sensitive to the notion that Amazon has made a significant investment in EC2 and we do not wish to damage that investment in any way. Thus we decided that we would only read the API specifications that Amazon publishes free of charge and without restriction in designing Eucalyptus. Amazon, it turns out, does an excellent job of defining the EC2 specification since we get reports that the system is faithful in its ability to reproduce the EC2 functionality. Thus Eucalyptus is an implementation of the API, though, and not a reverse engineering of the EC2 architecture and implementation.

7) What are the technical strengths and weakness of a Eucalyptus 'cloud' as compared to competitive, commercially-licensed alternatives?

The strengths and weaknesses are probably the same. It is an open source infrastructure built from open source components. The strength of this approach is that it leverages development and innovation emanating from the open source community. The weakness is that tracking and identifying the rights set of innovations is labor intensive and time consuming. We spend a lot of time figuring out work arounds, bug fixes, new feature exploits, etc. for some of the components we include. A commercial system based on proprietary software controls all of the software in its code base. This control can be a strength if it is managed well or a weakness if it stifles innovation.

I guess what I'm saying so poorly here is that commercially-licensed systems and Eucalyptus each face their own challenges when it comes to quality and utility. We are very focused on the challenges we face as we want the users of Eucalyptus to share our enthusiasm for it. Commercial alternatives undoubtedly share this commitment for their own products. They just have different technical challenges to address.

8) How does deploying network services within a Eucalyptus 'cloud' provide advantages an enterprise over a more traditional server infrastructure?

Web services, by their very nature, must be tolerant of fluctuating availability and performance in the underlying infrastructure that implement them. At the same time, they must be able to operate at scale if offered load warrants the additional complexity needed to achieve scalability. More and more the combination of these two characteristics is becoming important in the enterprise.

9) How do you think Windows Azure fits into the cloud computing space?

Windows Azure takes a different and important new position with respect to cloud systems. Before Azure, clouds were almost exclusively service venues. By that I mean that the cloud provider offers a externally available service that, more often than not, is web accessible. Users access these services either by making requests (e.g. Amazon AWS) or by uploading code (e.g. Google's AppEngine). Azure is different in that it links the local programming environment to the cloud through a coordinated execution framework. That is, it blurs the line between where the public cloud ends and the "on premise" infrastructure begins.

There are tremendous advantages to the Azure approach but also potential obstacles. The primary advantage is that in Azure, the cloud is integrated seamlessly in the application. It will not automatically turn a non-cloud application into one that is well-suited to the cloud, but it does abstract away many of the cloud details associated with resource provisioning, failure modes, etc. The primary obstacle is that an Azure application is just that - an application that cannot function without the Azure cloud environment. One advantage of the service venue approach is that the service interface admits many different applications and usage models. This "separation of concerns" promotes modularity and re-use whereas having the cloud in the "fabric" of the application (as in Azure) limits this flexibility.

However, it is clear that the approach taken by Azure is one that certainly needs to be considered as it constitutes a distinct innovation with respect to the previous cloud offerings.

10) What does the future hold of Eucalyptus and cloud computing?

For Eucalyptus, our future will hopefully include a greater understanding of what cloud computing will mean for those wishing to deploy an "on premise" cloud. We believe that the future of cloud computing, in general, will have a significant hybrid cloud component in which users wish to combine their own resources with those offered in public clouds. We plan to investigate and develop new cloud features and abstractions that make the synergy between the "cloud you own" and the "cloud you rent" easier and more efficient to exploit.