Above The Cloud

Back at Mix09 there were a number of releases around the Azure Services Platform. One of them was an announcement about the upcoming release of the geo location feature – the ability to choose how your services and data is stored geographically. Well this feature has just reached our hands, being announced on the last day of April on the Windows Azure blog.

I thought I’d give it a spin since I found some spare time today. I created a simple service.svc with a single method and uploaded it to Windows Azure. One of the first things you will notice now during the creation of your project is the option to select your region.

You can be very specific about your location. Currently there are only two locations to choose from with no definite announcements around which locations will come next.

You can also group multiple services/storage projects into a group called an ‘Affinity Group’.

A group is affiliated with a geo location (region). When you create other service or storage projects you can choose any existing affinity group to associate that project with.

Affinity groups are more than just a way of making sure all your services and data are hosted in the same region. They actually provide a way of communicating to the Azure Fabric Controller that services and data are related and should be located near to each other in the data centre for optimal performance. As mentioned in previous posts of mine, the Fabric Controller is already equipped with the ability to provision resources on the fabric in relation to performance, security, and fault tolerance requirements of the user application. Affinity groups are the first step towards providing this control of deployment to the application owner.

As just mentioned there are two locations currently: USA – Northwest and USA – Southwest.

Selecting Northwest first, I uploaded my simple service and performed a name-server lookup:

I also tried a trace route however was unable to get any responses deeper than a router on msn.net. I wrote a quick console application to talk to my cloud service. It calls the service a thousand times sequentially, recording the time taken. In fact I ran this test twice, and the 2 results (in milliseconds) are 717138 and 726795 respectively, the average being about 722 seconds.

Next I ran the same test after creating my project in the southwest region instead. Here’s the lookup:

Interestingly the 2 locations are on the same class B network. This is a little suspicious and I’m interested in hearing from someone in the Azure team as to why this is. I would have thought the range of IPs would be a lot more diverse. I suppose one of the things about minimizing the number of machines required means also minimizing the number of IP addresses needed. Perhaps cloud computing will solve the IP4 empty pool problem?

As before I ran the console application twice and turned up the following two times (in milliseconds) – 767903 and 751676 with the average being about 760 seconds. As you can see this is about 40 seconds slower than when the service was hosted in the Northwest.

As an Australian, I thought I’d run these tests just to see which is the better option. And even though Southwest US is geographically closer to Australia than Northwest US, it would appear that Northwest US is a better choice for your Azure applications if you are targeting Australian users.

While playing around with this new feature I discovered a hiccup. Previously all services were located in one location: USA Northwest. Selecting Southwest for the first time resulted in my production instances not starting up. They simply sat on initializing for an extended amount of time. I stopped and restarted several times and deleted/recreated my project, a few times as well before they finally booted up completely, allowing me to run my tests.

I’m looking forward to similar releases in the future around update and fault domains as we see the full vision of the Azure Fabric come to fruition.

A few weeks ago I discussed the concept of what the Azure Fabric really is, and then followed with a discussion about how the Azure Fabric Controller works to manage the fabric. In this post I’d like to dive a little deeper into the lifetime of a server in the Azure Fabric and the process of deploying resources on demand.

Server Utilization

In many data-centres server utilization at any given point in time is quite low. The ultimate goal of a lot of cloud computing providers is to provide cost savings to customers by increasing server utilization, which reduces the number of servers required. Microsoft is no exception, and with Azure hopes to maximise server utilization through multitenancy with the appropriate security boundaries and without impacting on performance. Their goal is to improve the performance per watt per dollar ratio by running more services on less hardware.

In Azure the security isolation is managed via virtualization. A virtual machine dedicated to any application instance ensures it is isolated and also provides some benefits around health, monitoring and performance management. And of course, this provides the correct abstractions from the host operating system and hardware.

Obviously Hyper-V plays a major role in all of this. Azure uses versions of Server 2008 for its host machine and virtual machine environments. These versions are much like the versions you see today except that there are optimizations for running in the Azure Fabric. In a Hyper-V scenario, the host operating system sits above the hypervisor just like the guest virtual machines.

What this means for Azure is that the host and guest operating systems can be provisioned in the same way. Currently this happens by VHD (virtual hard disks). The VHDs are created and managed offline. When a new server is available and needs to be utilised, a host VHD is loaded onto the server, followed by guest VHDs as necessary. The host VHD will be Hyper-V enabled, and the machine will be booted with that host OS (a feature made available in Server 2008 R2 and also coming to Windows 7).

Servicing the images also happens offline. If a patch or service pack needs to be applied, it is difficult to apply it to thousands of virtual machines in a data centre all at once. It is also risky to the applications that are already running on those instances. Instead, the images are updated offline and when a new machine needs to be provisioned, the host or guest VHD’s it receives will already be patched and updated.

So how does this deployment happen?

Let’s emulate the life cycle of a server. This could be any server hardware; the key is that it is empty – no host operating system, just a network bootable bios enabled.

The first thing that happens is the server remote boots a maintenance OS – something similar to WinPE. There is an agent on this OS that knows how to find and talk to the Azure Fabric Controller. The FC then tells the agent to prepare a host partition. The agent then downloads a Hyper-V enabled version of Windows Server 2008 Core (since the host OS should be as lightweight as possible). This VHD is read-only and only contains the basics. Then a differencing disk is attached to the base VHD where all changes will be made. This provides the advantage of being able to rollback to a known state at any point in the server lifecycle. Images that are downloaded are really nothing more than XCOPY.

At this point the maintenance OS shuts down and the server reboots into the Server Core OS that has just been setup. There is also an agent on this OS that can talk to the Fabric Controller to get its next instructions. That next instruction might be to download a new virtual machine to run on this server. A Windows Server 2008 Enterprise VHD is downloaded (could be any version of server 2008) and as with the host OS this base VHD is read only, with a differencing disk downloaded and attached as the next step. Further to that an application specific VHD is downloaded – this contains specific applications required by your application, such as IIS, .Net Framework, etc. Finally your actual application is downloaded – this might be your web role, and this whole guest partition will serve that role.

Next the host OS agent receives a message to provision another virtual machine. It creates the guest partition and this time downloads a Server 2008 Core VHD instead (since the application that will be running doesn’t need all the features of Enterprise edition). As before, a differencing disk is applied to the partition and an application VHD is attached as well before the actual application is downloaded to the box.

The host OS agent receives another message from the Fabric Controller – another Server 2008 Enterprise application needs to be deployed. Since the first partition also was Server 2008, and the base VHD is read only, that VHD can be shared by the new partition we are creating. Not copied: shared. This saves on disk space for that particular server (since the Enterprise VHD is huge).

Given the nature of the read only base images for each guest (and the host) it is very easy to roll back an instance to a safe point in time without having to recreate the entire partition or reimage the whole server.

Because it is possible that more than 1 server might want to get a copy of a VHD at any time, the VHDs are sent using a multicast protocol, meaning that only 1 copy of the image is being sent at any given time, and multiple host agents can receive the image at any given time. Naturally this optimizes internal bandwidth.

This Sounds Great – Can I Do This In My Own Data Centre?

There’s been gossip about whether or not Azure will be something you can run in your own data centre and to date the official word has been ‘no’. However the innovations we see in Azure have started to filter their way down into Microsoft products and technologies, such as Windows Server, Hyper-V, and Virtual Machine Manager.

Pretty soon it will be possible to manage your own cloud infrastructure using the above Microsoft products.

The Azure Fabric Controller (FC) is the service which monitors, maintains and provisions machines to host the applications that we (the developer) create and store in the Microsoft cloud.

Previously I’ve helped define the word ‘fabric’ and in specific, discuss the details of the Azure Fabric and the Development Fabric. The Azure Fabric Controller is responsible for managing all the nodes and edges in the Azure Fabric, which is essentially servers (both provisioned and not), load balancers (usually hardware balancers), power-on automation devices, switches, routers, etc.

The Fabric Controller manages different devices in different ways. For example, hardware load balancers are supported through a driver model. Each balancer could be different hardware type, vendor, etc. Azure abstracts the communication by exposing the balancer through a custom driver for that specific model. However, how it manages powered-on servers is slightly different.

There is a special service that runs on all powered-on servers/instances and the Fabric Controller communicates with the server via this service. The service tracks two things: the ‘current state’ of the server and the ‘goal state’. A goal might be to run a worker instance, or it might also be to remain idle as part of the free inventory. The current state might be something like ‘initialising’ or ‘idle’. The Fabric Controller and the local service can then manage how the system gets to the goal state from the current state.

When an error occurs, the service detects the fault and changes the current state accordingly (something like ‘faulted’). Once again the Fabric Controller and the service can manage what’s required to get back to the goal state. This might mean a reboot, or perhaps reprovisioning the whole server. The Fabric Controller can take alternative options like provisioning another resource to host your instance.

This mechanism is quite useful in that repeat patterns of failures or hardware faults can be easily identified and a server can be marked as ‘inoperable’.

One of the key roles of the Fabric Controller is to provision resources based on the needs of the applications written by the developer. To manage this it has a declarative service model that defines exactly what is needed by the application.  This model covers things like what roles the application performs and how those roles communicate, what operating system requirements there are (does it need IIS for example), how much CPU is needed, bandwidth required, etc. It can even specify what guest operating system to use, and if a dedicated box is required or if virtualisation is enough.

There is also some redundancy tolerance at the provisioning level, referred to as ‘fault domains’ and ‘update domains’. For example, you can specify that a particular application be distributed over 3 fault domains, meaning that your application will be located in different parts of the fabric such that server or switch failures will only bring down 1 instance. The Fabric Controller can model a certain amount of risk to sections of the fabric based on areas of single point of failure, and it uses these statistics when deploying your application into the fabric. This also applies to ‘update domains’ which essentially ensure that system updates that take services offline will affect your application 1 piece at a time, meaning you can ensure continuous availability.

When it comes time to provision a resource for your instance (being 1 instance of 1 role), the Fabric Controller will examine those specified requirements, and look through its inventory (fabric) for a resource that matches. It then changes the Goal State of the node and the provisioning process begins.

At this point you might be saying: “Hey, I can’t configure any of this stuff right now you liar!”. And you’d be right: as the developer we don’t yet have this fine grained control over how the Fabric Controller manages our applications. For the CTP, what we have been given is some templates instead, known to you and I as the Web and Worker Roles. However on the Azure side, these templates are translated into some predefined specifications around fault and upgrade domains, software requirements, and machine level resources. For example, these will always be a Server 2008 Enterprise running x64, 1.7Gb of RAM, and 250Gb disk space. In the future we will see more specific control become available, specifically to organisations who pursue an SLA route with Microsoft. We should see some of this later in the year (2009).

The Fabric Controller itself is highly redundant, with 5 to 7 replicas being available at any given time. The state of all the nodes in the fabric is replicated across all of these replicas to ensure that no matter which Fabric Controller is managing your particular node, its state tracking is 100% up to date. In the event that all Fabric Controller nodes go down, all existing services will still continue to run. However the provisioning and fault tolerance aspects will obviously be offline.

What I find really interesting is that the Fabric Controller replicas are all managed by a miniature version of Azure as well. This means that there is a service definition for the “Azure Fabric Controller” application which is deployed as a set number of instances, and has support for all the same kinds of fault and update domains. A new fabric controller can be provisioned automatically should there be failures, etc.

That’s about all I wanted to cover with the Fabric Controller. There’s a lot more to learn about how Azure manages its infrastructure, especially around deployment of host images and virtualised guests’ images. A future post perhaps.

Last Friday 13th March 2009, Azure suffered from some pretty catastrophic events. The full details of what happened can be found here on the Windows Azure blog. The key point was that an operating system upgrade caused a bunch of servers to fail.

What’s interesting in that article is this point:

the Fabric Controller automatically initiated steps to recover affected applications by moving them to different servers.  The Fabric Controller is designed to be very cautious about taking broad recovery steps, so it began recovery a few applications at a time

Fantastic! Why do I think that?

Well, the fabric controller is designed to handle individual server fail over. When a server hosting your instance fails, it needs to spin up a new server/instance to replace it, all without you, the user, doing anything. What would happen if for some reason the fabric controller thought that EVERY server was unavailable? Would it simultaneously attempt to re-spin up a new server/instance for every single existing server as quick as possible? What would that do to the controller? What if all new instances it was spinning up were failing too? Would it flat-line the CPU?

I think you get the gist. Anyway, this is why we have CTP. Better to discover these issues now, rather than RTM.

Also, Steve Marx posted some information around the event and the communication failures on his part during the down time. Well done Steve for being so honest.