Just a quick post to point out two interesting (but unrelated) cloud links I stumbled upon today.
The first is the Cloud Computing Manifesto which exists to facilitate the discussion between the major vendors to come up with a set of key principles to be adhered to by all providers. The site also describes a Cloud Computing Bill of Rights and has been constructed in direct response to this post by Steve Martin regarding a secret cloud manifesto written without input from his employer, Microsoft. After a cursory inspection, it seems very lightweight to me; hopefully they will pad it out soon because the effort has merit.
The other interesting thing I discovered today was the Cloud Computing Podcast. I had toyed with the idea of creating such a podcast myself; luckily I committed absolutely no resources to doing so! I’ve listened to a couple now and the premise of the podcast seems to be around reacting to cloud specific industry events and articles. It’s not overly long, each episode around 15 minutes. But they are very regular, averaging around two a week. Also, there’s a long backlist to listen to since its been going since October last year! There’s definitely not enough Azure coverage, but hey, not everyone’s perfect. I’m definitely adding this one to my RSS feeds.
The Azure Fabric Controller (FC) is the service which monitors, maintains and provisions machines to host the applications that we (the developer) create and store in the Microsoft cloud.
Previously I’ve helped define the word ‘fabric’ and in specific, discuss the details of the Azure Fabric and the Development Fabric. The Azure Fabric Controller is responsible for managing all the nodes and edges in the Azure Fabric, which is essentially servers (both provisioned and not), load balancers (usually hardware balancers), power-on automation devices, switches, routers, etc.
The Fabric Controller manages different devices in different ways. For example, hardware load balancers are supported through a driver model. Each balancer could be different hardware type, vendor, etc. Azure abstracts the communication by exposing the balancer through a custom driver for that specific model. However, how it manages powered-on servers is slightly different.
There is a special service that runs on all powered-on servers/instances and the Fabric Controller communicates with the server via this service. The service tracks two things: the ‘current state’ of the server and the ‘goal state’. A goal might be to run a worker instance, or it might also be to remain idle as part of the free inventory. The current state might be something like ‘initialising’ or ‘idle’. The Fabric Controller and the local service can then manage how the system gets to the goal state from the current state.
When an error occurs, the service detects the fault and changes the current state accordingly (something like ‘faulted’). Once again the Fabric Controller and the service can manage what’s required to get back to the goal state. This might mean a reboot, or perhaps reprovisioning the whole server. The Fabric Controller can take alternative options like provisioning another resource to host your instance.
This mechanism is quite useful in that repeat patterns of failures or hardware faults can be easily identified and a server can be marked as ‘inoperable’.
One of the key roles of the Fabric Controller is to provision resources based on the needs of the applications written by the developer. To manage this it has a declarative service model that defines exactly what is needed by the application. This model covers things like what roles the application performs and how those roles communicate, what operating system requirements there are (does it need IIS for example), how much CPU is needed, bandwidth required, etc. It can even specify what guest operating system to use, and if a dedicated box is required or if virtualisation is enough.
There is also some redundancy tolerance at the provisioning level, referred to as ‘fault domains’ and ‘update domains’. For example, you can specify that a particular application be distributed over 3 fault domains, meaning that your application will be located in different parts of the fabric such that server or switch failures will only bring down 1 instance. The Fabric Controller can model a certain amount of risk to sections of the fabric based on areas of single point of failure, and it uses these statistics when deploying your application into the fabric. This also applies to ‘update domains’ which essentially ensure that system updates that take services offline will affect your application 1 piece at a time, meaning you can ensure continuous availability.
When it comes time to provision a resource for your instance (being 1 instance of 1 role), the Fabric Controller will examine those specified requirements, and look through its inventory (fabric) for a resource that matches. It then changes the Goal State of the node and the provisioning process begins.
At this point you might be saying: “Hey, I can’t configure any of this stuff right now you liar!”. And you’d be right: as the developer we don’t yet have this fine grained control over how the Fabric Controller manages our applications. For the CTP, what we have been given is some templates instead, known to you and I as the Web and Worker Roles. However on the Azure side, these templates are translated into some predefined specifications around fault and upgrade domains, software requirements, and machine level resources. For example, these will always be a Server 2008 Enterprise running x64, 1.7Gb of RAM, and 250Gb disk space. In the future we will see more specific control become available, specifically to organisations who pursue an SLA route with Microsoft. We should see some of this later in the year (2009).
The Fabric Controller itself is highly redundant, with 5 to 7 replicas being available at any given time. The state of all the nodes in the fabric is replicated across all of these replicas to ensure that no matter which Fabric Controller is managing your particular node, its state tracking is 100% up to date. In the event that all Fabric Controller nodes go down, all existing services will still continue to run. However the provisioning and fault tolerance aspects will obviously be offline.
What I find really interesting is that the Fabric Controller replicas are all managed by a miniature version of Azure as well. This means that there is a service definition for the “Azure Fabric Controller” application which is deployed as a set number of instances, and has support for all the same kinds of fault and update domains. A new fabric controller can be provisioned automatically should there be failures, etc.
That’s about all I wanted to cover with the Fabric Controller. There’s a lot more to learn about how Azure manages its infrastructure, especially around deployment of host images and virtualised guests’ images. A future post perhaps.
To answer that we first have to look at the term ‘fabric’.
You can think of a fabric (or ‘switched fabric’) as a network of interconnected nodes. The connection is facilitated by high-speed switches through fibre-optic connections, while nodes are usually servers and other technical infrastructure. 
If the network is large enough and viewable from a distance, it could look like a ‘weave’, or a ‘fabric’. This terminology is also used at the processor level describing the network of buses on the chip.
The Azure Fabric is not much different. It is made up of nodes, which are servers running Windows Server 2008 and load balancers, while the edges are the power, Ethernet, and serial communications. Each computer node might be a dedicated server, or a server petitioned for multiple virtualised environments.
When we write our Azure applications locally, there is a simulation of this fabric running on our machines, called the ‘Development Fabric’. This lets us simulate load balancing and multiple nodes for our web and worker roles.
The Development Fabric is facilitated via a series of running processes that simulate load balancing and hosting your application.
When you run a project of type ‘Cloud Service’ (the project that contains a service definition and Role links out to other projects), the development fabric is initiated (as well as a simulation of local development storage). In the diagrams below, we see the process tree as initiated by Visual Studio and the location of the files.
It is actually possible to spin these services up yourself, rather than needing Visual Studio to do it. Although I haven’t actually tried, it should be possible to copy these files to a testing server and run them up so that you can hand your application over to your testers.
Another thing you might be able to do (I have yet to try this as well) is spin up the load balancer and various services inside unit tests. Most of these executables are in managed code and trusty old Reflector can bust them open for us. Looking at the code for DFloadbalancer, we see that it is simply simulating a load balancer through usage of WCF, binding via a Named Pipe. We are given the load balancer classes in the LoadBalancer.dll so I’m confident there is some cool unit test automation to be achieved via directly calling the necessary classes. If you do go down this track, please let me know how far you get.
In a future post I will be discussing the Azure Fabric Controller.
Recently we were discussing pricing of cloud hosting versus traditional ‘shared hosting’ and ‘dedicated server hosting’ (in an internal work mail thread). As a consultant, I dabble in my own domains and hosting, and have often sought out the cheapest option for what I needed. This has bitten me before, having services hosted with US hosting companies with no accountability, and I now host stuff (like this blog) on Australian servers.
But I thought it somewhat hypocritical to have a blog about the cloud and not host it in the cloud as well , so I started to investigate pricing with the various providers like GoGrid and EC2. I decided that I would be more than happy to go through the pain of moving from my shared host if I could get a single virtualised server that I could throw all my domains at, for around the same price or less.
But the more I investigated, I just couldn’t see the pricing working in my favour – not for my single server scenario where scalability is not really required (I need a few more hits yet – tell your friends!). I made my calculations based on the lowest end server available, with minimal bandwidth and storage. I estimated 730 hours in a month on average. Here’s the numbers:
| Host | Per Hour | Per Month |
| My Shared Host | - | $24.95 |
| Amazon EC2 | $0.125 | $90.00 |
| GoGrid | $0.19 | $138.70 |
Ok I realise we’re not comparing apples with apples here. Each service offers differing functionality and is really designed to offer economies of scale, whereas my shared host is designed for my exact market.
So if you are looking at entry level, I guess the point is really: don’t bother.
That being said, what about at the other end, where your company can benefit from scalability and other distributed aspects of cloud computing? Well I have no answer to that, however this post on Google groups presents some interesting arguments around why cloud computing may never actually be viable. Its worth the read.
Unfortunately I’m not at Mix09. But I doubt that the attending delegates would get as broad a view across everything coming out of Mix as you and I sitting at home on our computers, watching the blogs and the screen casts, will. There’s a lot happening, in many different spaces, across many different teams. While the Azure team are good at announcing stuff via twitter and their blog, there’s still some stuff that is cloud related and not necessarily related to utility computing (reminding you that my definition of cloud may not be the same as yours). I’ll attempt to cover it all in case you missed the tweets and couldn’t be bothered reading 100 different blog posts.
On to the list! (in no particular order)
Lots of good stuff has come out of Azure. Some of it we knew was coming, some of it we expected to come. There’s also been some announcements about the next phase of work and what’s to come.
Quite simply, you can now write code that will run in full trust, which of course means running native code (PInvoke). This applies to both web and worker roles thankfully and means you can use 3rd party libraries that depend on full trust. The only thing to be aware of is that the roles themselves still run under a user account with limited privileges. You still can’t touch the registry, etc. This is necessary for security (obviously). You don’t want user code to hit the windows service needed by the fabric controller to monitor an instance (lest we employ hacks to get free hosting hours!).
To enable full trust, a simple config change to the service definition is required. More info here.
Where there is the ability to support native code, comes the ability to support other languages. Enter: FastCGI. Ever attempted to host that Wordpress blog on IIS7? You’d already be familiar with FastCGI then. This will enable PHP applications to run in Azure. It requires some work to setup, but there is a new web role for FastCGI to get you started quicker.
If no one else has ported Wordpress to Azure by Sunday, I’ll do it myself. More info here.
Previously Azure was hosted out of a data centre in Seattle (I think, please correct me if wrong). Although Microsoft has data centres all over, they limited Azure to this location to begin with. This is now changing however with a second location in the country’s south. As the technology around Azure improves to support locations, the ability to provision the software on new (existing) data centres has (nearly) become available: essentially it is now scalable to n data centres.. You’ll be able to group your applications and define how they will be geospatially dispersed. This lends itself to better performance for your web visitors, as well as having other benefits.
This is a ‘coming soon’ technology more so than ‘here right now’. More info here.
Previously it was a little hard to get across all the bits and pieces needed to get going in the Azure world. Now there is a dedicated developer site to get us moving which includes a lot of content that was previously floating around in different locations (SDKs, blogs, etc). It even has a clearer path for new users to get started with Azure around the token redemption process (something that caused a lot of problems previously). Also there was recently an update to the Azure SDK that saw the VS tools merged and an auto-update process to make things just that little bit smoother.
Yes its true, all this Azure talk will be well complimented with some cloud compute power via GoGrid. All delegates get $200 worth of hosting time.
If anyone doesn’t want to use their credit, please donate it generously to this blog so I can review it for you.
These are some things I’ve picked up from the live twitter feed (mostly language related):
So, launch is at the end of this year?
Ever had to implement the delegated authentication story to get access to someone’s messenger contacts? Previously its been a pain. The API’s have changed continuously, the tooling hasn’t been fantastic, the merge with Azure has made lots of previous documentation redundant, you get the idea. Well now life is about to get a lot easier with a dedicated toolkit called ‘Windows Live Messenger Web Toolkit’.
See the Live Services Blog for the low-down. Find the kit here: http://dev.live.com/messenger/
If you haven’t heard (or didn’t see it coming) Silverlight 3 beta is now released. Of course there’s various enhancements, bug fixes, etc. But the most cloud focused aspect is this ability to run Silverlight applications on the desktop, similar to a certain Adobe product.
Of course you could speculate that this negates the need for WPF but lets be honest, WPF is way too cool for that to ever happen, and WPF supports the full framework, whereas Silverlight is (or was) a 4mb plugin. Still, install your app to the cloud, let it run on the desktop. Pretty cool eh?
Read here.
This is an interesting concept and I’m not sure if it will take off (time will tell, don’t worry I’ll edit this post if it does). The idea is that you can create a special installer for your product and upload it to the Web App Gallery where other people can install it in a similar fashion to Click-Once. I guess the motto here is store your web app installer in the cloud. Coupled with this announcement is that MVC is now RC (at the bottom of the deep blue – sea sea).
Ok truthfully the gallery (and installer) have been around for more than a few hours, but really, its the updated support via IIS extensions, MVC, and new installer release that really make the Mix09 announcement.
Read more here and here.
Well of course not. There’s lots of stuff being announced that may or may not relate to this blog. But hey, lets hope they save something for Tech Ed and PDC right?
I’m still watching for some Mesh announcements although it looks like they are simply demoing some of the functionality I’ve already talked about in relation to the Live Framework Client.
Last Friday 13th March 2009, Azure suffered from some pretty catastrophic events. The full details of what happened can be found here on the Windows Azure blog. The key point was that an operating system upgrade caused a bunch of servers to fail.
What’s interesting in that article is this point:
the Fabric Controller automatically initiated steps to recover affected applications by moving them to different servers. The Fabric Controller is designed to be very cautious about taking broad recovery steps, so it began recovery a few applications at a time
Fantastic! Why do I think that?
Well, the fabric controller is designed to handle individual server fail over. When a server hosting your instance fails, it needs to spin up a new server/instance to replace it, all without you, the user, doing anything. What would happen if for some reason the fabric controller thought that EVERY server was unavailable? Would it simultaneously attempt to re-spin up a new server/instance for every single existing server as quick as possible? What would that do to the controller? What if all new instances it was spinning up were failing too? Would it flat-line the CPU?
I think you get the gist. Anyway, this is why we have CTP. Better to discover these issues now, rather than RTM.
Also, Steve Marx posted some information around the event and the communication failures on his part during the down time. Well done Steve for being so honest.
Virtual Earth is one of those Live Services that most people have probably heard about. It is extremely easy to introduce VE style maps into your application via the controls found in the Live SDK. Just drag and drop a server side VE control onto your ASP.NET application and its done!
In the past I’ve blogged about competitions that I’ve heard about on my other blog. This one comes from Craig Bailey on Twitter.
http://www.craigbailey.net/live/post/2009/03/15/Virtual-Earth-Developer-Contest.aspx
Essentially you don’t even need a completed application to show. Just demo something cool via video that is using Virtual Earth and you’re in the running. Or even just write an article about a new concept that utilises the technology. For example, you might have an idea for a JavaScript game that uses the VE surface as a backdrop.
So get the creative juices flowing. Apparently you get a free T-shirt just for submitting an entry!
I’ve often wondered what the complete list of services are from the Live Services brand.
Well recently I was looking through the Terms and Conditions for Live Services (yes I had nothing better to do at the time) and discovered in the appendix this list of services and reference material that I thought I would share (below). What’s not covered in this list (as far as I can see) is Popfly and Virtual Earth.
Windows Live ID Delegated Authentication service, available at:
http://msdn2.microsoft.com/en-us/library/cc287637.aspx
Windows Live ID Web Authentication service, available at:
http://msdn2.microsoft.com/en-us/library/bb676633.aspx
Microsoft Federation Gateway, documented at:
http://msdn.microsoft.com/en-us/library/cc287610.aspx
Windows Live Messenger Library documented at:
http://msdn2.microsoft.com/en-us/library/cc298458.aspx
http://msdn2.microsoft.com/en-us/library/cc303092.aspx
Windows Live Messenger IM Control documented at:
http://msdn2.microsoft.com/en-us/library/bb936682.aspx
Windows Live Presence API, available at:
http://msdn2.microsoft.com/en-us/library/bb936691.aspx
Live Search API service is documented through the WSDL file at:
http://soap.search.msn.com/webservices.asmx?wsdl
Windows Live Photos API service documented at:
http://msdn2.microsoft.com/en-us/library/bb932323.aspx
Windows Live Photos Control, available at:
http://dev.live.com/photocontrol/
Windows Live Application Based Storage service documented at:
http://msdn2.microsoft.com/en-us/library/cc305108.aspx
The Silverlight Streaming by Windows Live documented at:
http://msdn2.microsoft.com/en-us/library/bb851621.aspx
Windows Live Contacts API documented at:
http://msdn2.microsoft.com/en-us/library/bb463989.aspx
Windows Live Contacts Control, available at:
http://dev.live.com/contactscontrol/
Windows Live Alerts API documentation can be found at:
http://msdn2.microsoft.com/en-us/library/bb259752.aspx
Intro to Windows Live Spaces API documented at:
http://msdn.microsoft.com/en-us/library/bb447732.aspx
Windows Live Spaces MetaWeblog API documented at:
http://msdn.microsoft.com/en-us/library/bb259702.aspx
Application Catalog, available through:
http://lx.azure.com
Live Framework APIs, available at:
http://go.microsoft.com/fwlink/?LinkID=130293
Please note that I cannot validate the authenticity of any of the above links.
It occurred to me that I had not really defined the concept of "cloud" although many of you already understand the meaning behind the term. Still, it makes sense that I spend a small amount of time defining what it means to me, and therefore, how it will impact the content of this blog.
Wikipedia defines ‘cloud’ as follows.
The term cloud is used as a metaphor for the Internet, based on how the Internet is depicted in computer network diagrams, and is an abstraction for the complex infrastructure it conceals
You know I’d always seen the cloud depiction (even used it in some Visio diagrams) but my lazy brain never made the association. Sounds somewhat reasonable now.
So what do we mean when we think of services hosted in the cloud? Well, consider for a minute something old-school like billboards or IRC. Here were theses centralized services that offered something, free or otherwise, that provided value to its consumers. We liked to chat to other people, IRC facilitated this via a common protocol. We used "rich" clients (mIRC?) to make the experience even more enjoyable, simpler, and more automated.
Would we consider IRC to be a cloud offering? Its not really replacing any service locally (except standing up and talking to your neighbor), but that seems to be a common concept around cloud – moving the work from our desktops/servers onto the Internet in a way that we now no longer hold any responsibility over maintaining the app or its infrastructure needs.
Now consider something like Office Communicator. My employer has a communicator server so that we can all message each other. But I’ve been using MSN since before I can remember to achieve the same thing. How does this fit into our definition of cloud based services?
The more I think about it, the more examples I come up with. Email is a classic. My organization has an Exchange server, but I’ve had a Hotmail account since the dawn of time. More recently, we store our contacts in Facebook, our professional contacts in LinkedIn, and our calendars in Google.
Could it be that this is in fact nothing new? Could the cloud terminology perhaps be just hype? I decided to delve into some other people’s opinions, and here’s what I discovered.
Finding definitions about ‘cloud computing’ is easy, but often relates to scalable infrastructure: something developers aren’t always very interested in.
So the question again is: what is this blog all about? I guess I could abuse the generality of the definition of ‘cloud’ to incorporate anything Internet related, but I really don’t want to do that. Instead, I want to focus on how cloud based applications are/will replace/change the way we do things today. For example I recently posted about Mesh. I see Live Mesh as being something that will change the way we fundamentally work with computers/phones, so it definitely deserves a mention. And cloud computing topics such as AWS, Azure, and GoGrid are always going to receive a special focus, of that I am sure.
Beyond that, I am open to suggestions. If you find I am straying too far, please let me know. If I am becoming to focused (I’ve got my blinkers on), again, let me know.
Straight to the point: if you haven’t signed up for the Mesh beta, go do it now!
So what is Mesh?
Think of Mesh (for now) as a synchronization platform. In Mesh you define folders, and you can sync those folders on 0 or more devices (such as computers, phones, etc). I say “0 or more” because you can just use the web interface if you desire, giving you access to your folders from anywhere.
I don’t want to go into what Mesh is and how to use it, so I’ll refer you to some links:
So why am I here?
What I really want to talk about is what’s coming next. You may or may not be aware, but there are in fact two Mesh’ (Mesh’s? Meshes? Mesheses?). Live Mesh Beta is available for anyone to use, however there is also a Live Mesh CTP. The latter is a completely separate Mesh (you cannot share folders across both). This second Mesh represents the ongoing development around Mesh. Users can still get practical usage out of the existing Mesh Beta without risk of the new development effort impacting them.
So what’s different?
Well for starters, access. The Beta is available for all, however you have to apply for a ‘token’ to get access to the CTP (invitations are managed through Microsoft Connect).
Functionally, they work much the same. There is a web desktop, and a client to download locally. Something of interest: the Mesh CTP client is actually called ‘Live Framework Client’. I won’t speculate as to what this might mean going forward (ok a little: the client will be more than just folder sync I’m sure, as we’ll see in a minute). The Mesh is referred to as the Live Framework Sandbox in the CTP version.
The functionality that you are familiar with is the folder sync, the ‘News’, and the ability to remote control to your various devices. In the CTP version, folder sync and remote access is disabled. I’m unsure how long this will last, but for now you can do these things in the Mesh Beta so its not a big concern.
So what’s the point?
This:
Applications! Notice the difference in the top bar from the other screen shot above of Mesh Beta? The ‘Apps’ tab lets you add applications to your Mesh desktop. In this case I’ve added an instance of ‘Corkboard’ and an instance of ‘MeshLists’ to my desktop.
This is what a lot of people have been expecting for a while now. Fully fledged applications running inside Mesh. And the best thing is that its not just constrained to the Web Desktop. When my Live Framework Client syncs, I get an icon on my Vista desktop. This icon is suspiciously named the same as the application I added to my Web desktop.
When I double click it:
The same application is now running on my desktop. And in fact, the same application now runs on my laptop as well, because it also has the Live Framework Client.
If you have access to the Mesh CTP, you may or may not be aware that the latest Azure SDK has a bunch of tutorials around Live Services, including the new Mesh CTP. It demonstrates using Live Delegation Consent to get access to a requesting user’s personal data in Live (such as messenger contacts, Mesh News, etc). Go check it out!
Summary
That’s a brief overview of the Mesh CTP. You should apply for a token now and check it out, then let me know what cool apps you’ve built for it! I can’t wait until they let the folder sync happen so that we can start doing interesting things with our apps around shared data.
This is going to be a very interesting year.
More Options ...
Categories
Tag Cloud
Blog RSS
Comments RSS
Void 
Life 
Earth 
Wind « Default
Water 
Fire Light Bad Behavior has blocked 310 access attempts in the last 7 days.
Previous Entries
Last 50 Posts 
Back
