Above The Cloud

A few weeks ago I discussed the concept of what the Azure Fabric really is, and then followed with a discussion about how the Azure Fabric Controller works to manage the fabric. In this post I’d like to dive a little deeper into the lifetime of a server in the Azure Fabric and the process of deploying resources on demand.

Server Utilization

In many data-centres server utilization at any given point in time is quite low. The ultimate goal of a lot of cloud computing providers is to provide cost savings to customers by increasing server utilization, which reduces the number of servers required. Microsoft is no exception, and with Azure hopes to maximise server utilization through multitenancy with the appropriate security boundaries and without impacting on performance. Their goal is to improve the performance per watt per dollar ratio by running more services on less hardware.

In Azure the security isolation is managed via virtualization. A virtual machine dedicated to any application instance ensures it is isolated and also provides some benefits around health, monitoring and performance management. And of course, this provides the correct abstractions from the host operating system and hardware.

Obviously Hyper-V plays a major role in all of this. Azure uses versions of Server 2008 for its host machine and virtual machine environments. These versions are much like the versions you see today except that there are optimizations for running in the Azure Fabric. In a Hyper-V scenario, the host operating system sits above the hypervisor just like the guest virtual machines.

What this means for Azure is that the host and guest operating systems can be provisioned in the same way. Currently this happens by VHD (virtual hard disks). The VHDs are created and managed offline. When a new server is available and needs to be utilised, a host VHD is loaded onto the server, followed by guest VHDs as necessary. The host VHD will be Hyper-V enabled, and the machine will be booted with that host OS (a feature made available in Server 2008 R2 and also coming to Windows 7).

Servicing the images also happens offline. If a patch or service pack needs to be applied, it is difficult to apply it to thousands of virtual machines in a data centre all at once. It is also risky to the applications that are already running on those instances. Instead, the images are updated offline and when a new machine needs to be provisioned, the host or guest VHD’s it receives will already be patched and updated.

So how does this deployment happen?

Let’s emulate the life cycle of a server. This could be any server hardware; the key is that it is empty – no host operating system, just a network bootable bios enabled.

The first thing that happens is the server remote boots a maintenance OS – something similar to WinPE. There is an agent on this OS that knows how to find and talk to the Azure Fabric Controller. The FC then tells the agent to prepare a host partition. The agent then downloads a Hyper-V enabled version of Windows Server 2008 Core (since the host OS should be as lightweight as possible). This VHD is read-only and only contains the basics. Then a differencing disk is attached to the base VHD where all changes will be made. This provides the advantage of being able to rollback to a known state at any point in the server lifecycle. Images that are downloaded are really nothing more than XCOPY.

At this point the maintenance OS shuts down and the server reboots into the Server Core OS that has just been setup. There is also an agent on this OS that can talk to the Fabric Controller to get its next instructions. That next instruction might be to download a new virtual machine to run on this server. A Windows Server 2008 Enterprise VHD is downloaded (could be any version of server 2008) and as with the host OS this base VHD is read only, with a differencing disk downloaded and attached as the next step. Further to that an application specific VHD is downloaded – this contains specific applications required by your application, such as IIS, .Net Framework, etc. Finally your actual application is downloaded – this might be your web role, and this whole guest partition will serve that role.

Next the host OS agent receives a message to provision another virtual machine. It creates the guest partition and this time downloads a Server 2008 Core VHD instead (since the application that will be running doesn’t need all the features of Enterprise edition). As before, a differencing disk is applied to the partition and an application VHD is attached as well before the actual application is downloaded to the box.

The host OS agent receives another message from the Fabric Controller – another Server 2008 Enterprise application needs to be deployed. Since the first partition also was Server 2008, and the base VHD is read only, that VHD can be shared by the new partition we are creating. Not copied: shared. This saves on disk space for that particular server (since the Enterprise VHD is huge).

Given the nature of the read only base images for each guest (and the host) it is very easy to roll back an instance to a safe point in time without having to recreate the entire partition or reimage the whole server.

Because it is possible that more than 1 server might want to get a copy of a VHD at any time, the VHDs are sent using a multicast protocol, meaning that only 1 copy of the image is being sent at any given time, and multiple host agents can receive the image at any given time. Naturally this optimizes internal bandwidth.

This Sounds Great – Can I Do This In My Own Data Centre?

There’s been gossip about whether or not Azure will be something you can run in your own data centre and to date the official word has been ‘no’. However the innovations we see in Azure have started to filter their way down into Microsoft products and technologies, such as Windows Server, Hyper-V, and Virtual Machine Manager.

Pretty soon it will be possible to manage your own cloud infrastructure using the above Microsoft products.

When I was searching for an appropriate definition of ‘Multitenancy’ to quote, I found that people and organisations had differing opinions of what it really means. Even the Wikipedia article claims that different organisations are ‘using it as source of competitive differentiation’.

This could be true, but perhaps the term is just evolving. In a few short years the cloud landscape (ok lets call it the ‘sky’) has changed quite a lot with numerous vendors entering into this growing, competitive market. The original terminology has been abstracted and reapplied to new forms of virtualization.

If you scroll back through the calendar a few years I think there would have been a general consensus of what the term means. Gianpaolo from Microsoft once stated that in ‘a pure multi-tenant architecture a single instance of the hosted application is capable of servicing all customers (tenants)’. The key idea is that an application is deemed multi-tenant if it can service multiple clients in a unique fashion from a single instance of the application.

A good example that comes to mind is DotNetNuke. It is a CMS system that supports multiple unique portals. Each portal can be stand-alone and secured separately from one another, while still leveraging a single code base and database. It is the configuration that allows the separation.

A more SaaS example is salesforce.com where you can perform customer relationship management from the cloud. Every salesforce client is secured away from other clients, each instance presumably sandboxed from another.

So why would the term ‘multitenancy’ change over time?

Lets think about what the term means in a literal sense: multiple tenants. In other words, there can be many residents at a multitenant location, just like a rental house tenanted by multiple people. So what if instead of a house we had a server? Every server can have multiple applications installed – file services, DNS, Active Directory, DHCP services, etc. To me, that sounds like a server has multiple tenants.

What about virtualization? Multiple virtual machines running on a single host? Sounds like multiple tenants to me. In fact when you think about it in such simple terms, then anything could be multitenant. My head has billions of hairs – its multitenant. My street has many houses – its multitenant. The Azure Fabric has many servers – its multitenant. Each server has multiple virtual machines – they are all multitenant.

In summary, when Microsoft or any other company refer to something in their technology stack as ‘multitenant’ then there’s a fairly strong chance that it really is. In my opinion, the Wikipedia article on multitenancy needs some revision.