It’s been a while…but a lot has happened!

Can’t believe it’s been a whole year since I blogged anything!

And what a year it’s been!

This year has turned out to be one of the more interesting ones in my career – lots of very cool stuff that’s been taking up all my time…now that it’s coming to a close, I’m going to write up some blog posts to talk about what has happened!

Daily work stuff

In my day job, I work for a software development company that builds accounting software which runs in the cloud. The flagship product – the one I work on – is a .NET rewrite of a very successful C++/ISAM desktop app. The rewrite began as a desktop application replacement so it was designed as such, but with the emergence of cloud technologies, we decided to make the app a hybrid and offer a rich client interface to a cloud service.

The basic premise was a good one – the architecture was well layered so we were able to stick the internet between the server and the client and get the thing to run. However, the devil in the details was performance – the desktop application, quite reasonably, expected to have a fast, fat connection to a local database, but that changed once we separated the UI from the server.

Over the last couple of years, we tried all the quick wins and got the app out, but performance was still sub-optimal and we needed to come up with some fundamental changes to break the problem.

At the start of this year, a couple of us was given the opportunity to go in with a sledgehammer and try to break the problem out. In six weeks we had a pattern for some drastic improvements and we’ve spent the better part of this year rolling that pattern out through the most performance sensitive areas.

Big win!

I’ll blog about some of the technical approaches we have taken to get these dramatic improvements, but it’s safe to say that they are not for the faint-hearted!

We actually rounded off the last year with 3 months of really creative work building the first cut of the MYOB API. Working with @H_Desai and building stuff with ASP.NET Web API was a pretty satisfying experience!

Open Source Mania

In my copious free time (ha ha) I decided to clean up and open source a whole bunch of stuff.

Amongst the weaponry that is now public are:

BrightSword SwissKnife

A  (mostly functional-programming based) set of utilities to help with everyday programming.

I’ve got a blog series on practical functional programming which centres around SwissKnife coming.

Warning: Contains Monads.

BrightSword Squid 

Rhino Mocks is a nice way to create anonymous types for interfaces but it is slow as all heck and a bit unwieldy, so I decided to write a fast library to generate DTOs with behaviours for general purpose use. MVVM patterns with Property Change Notifications work straight from the tin! And it’s 3 orders of magnitude faster than Rhino!

Contains lots of nasty nasty IL and Reflection.Emit!

BrightSword Pegasus

A reference library for CQRS and (Dynamic) Queue Centric Workflows based on Windows Azure. This thing is seriously cool and makes short work of writing extensible, scalable cloud applications.

Most of the stuff is pretty well tested – and I even set up my own automated build-pipeline to monitor source control, compile, test and upload to nuget. So all these libraries can be referenced and used through nuget.

Use the Nuget Packages. Fork the repos. Submit Pull requests.

The Cloud Architecture Talk

@MaheshKrishnan and I have been doing talks together for a few years now, and we spend a fair bit of our time idly gossiping about technology and trends. We observed that the best-practice software patterns for the cloud aren’t well articulated and well understood by the community at large, so we decided to build a talk around the most common patterns.

We built a application to demonstrate the patterns as well, and drew up some diagrams. Unlike most of our other talks, we demonstrate very little code in the talk itself, but the code is all open source.

We are both humbled and pleasantly surprised by its reception!

So far, we’ve given it at:

• The DDD! Melbourne Conference
• MYOB technical brown bag series
• Melbourne Azure Meetup & VIC.NET
• Singapore .NET Dev SIG
• TechEd Australia 2013

And we have been invited to give it at NDC London!

Yes. That’s right! We’re speaking at NDC London!

TechEd Australia 2013

This year’s TechEd was pretty special!

Went to the Gold Coast with @simonraikallen, @couchcoder and @shaun_a_wilde.

Hanging with @shaun_a_wilde and @couch_coder

With @MaheshKrishnan at the Microsoft stand

@MaheshKrishnan and I gave a slightly modified version of the Cloud Architecture Patterns talk focusing on the implementation aspects. It was pretty well received, and we got a fair bit of good feedback from the audience. Lively discussion and lots of questions are always a good sign that people paid attention to the material and that it was relevant.

Got to meet @linqpad – would’ve loved to have an hour pairing with him on some stuff to do with monads, but perhaps another day.

Met and hung out with the usual suspects…

Hanging with @RockyH…

… @drmcghee and @frankarr. Didn’t know at the time that this was the last TechEd Frank Arrigo was going to be at – so I’m glad we got this shot!

I won a 3D printer at the raffle, but wasn’t around to collect it so I lost it! I still get guys poking fun at me for this!

Our company had been selected as a case study for our use of Azure, and it was a pretty special privilege to represent them at the last, conference-wide Lock Note session.

It was made all the more special because the Microsoft CVP responsible for Azure – Scott Guthrie – was presenting the Lock Note, and I got a chance to get on stage with him and talk for five minutes about how we used Azure.

So the reason I wasn’t around to claim my printer was that I was hanging out with @ScottGu, rehearsing for the Lock Note, and I got to spend about half an hour just hanging with the Gu, who is, for the record, a very friendly, down-to-earth guy.

Pretty awesome, all told!

Migrating a Web Application to Azure: Step 4

Moving the application to Windows Azure

 

A New Project

The first step is to add a new cloud project to the solution.

Ensure that you do not add any web or worker roles.

We will convert the existing web application within the solution to a Web Role in the Azure Project.

Right Click on the ‘Roles’ folder in the Azure Project, select ‘Add’ and then ‘Web Role Project in solution…’.

Select the web application – in this case ‘SharPix_Step2’

After adding the project as a Web Role, it should look like this:

Right-Click on the ‘SharPix_Step2’ role and select ‘Properties’

In this dialog, under normal circumstances, you should set Instance count to at least 2 to enable fail-over and scaling, and you would pick an appropriate VM size.

Also, we will start by using the Compute and Storage Emulators, so specify ‘UseDevelopmentStorage=true’ as the Diagnostics storage account.

Running In Emulator…

Visual Studio comes with a tiny taste of the cloud built-in, in the form of the Storage and Compute Emulators.

Simply running our new Azure Project gives us:

Note the URL and the port number. This is no longer the Visual Studio Development Web Server, but the Compute Emulator in action. Examining the Windows Tray shows us that the emulators are running….

To the cloud!

In order to push the Azure Project to our cloud environment, we need to configure Visual Studio and set up our credentials to ensure that we carefully control access to our Azure environment.

Right-Click on the Azure Project and select ‘Publish’. We could use previously issued certificates here, or issue a new one for this project by selecting ‘Add’.

Create a New Certificate, and fill in the details required.

Visual Studio creates a new certificate and stores it in a temporary, obscure, folder. Copy the path from this dialog onto your clipboard and head over to the Windows Azure Management portal.

Click on ‘Management Certificates’, select the Azure Subscription you want to deploy to, and select ‘Add Certificate’ from the right-click context menu.

Once the certificate is uploaded, select and copy the Subscription ID from the portal and fill it into (3) of the Windows Azure Project Management Authentication dialog shown above. Give the Project some reasonable name and click ‘OK’

After dispatching all the other dialogs, you will be left with the first ‘Publish Windows Azure Application’ dialog. Ensure you have selected a Production Environment, and specify the storage account to use.

Hurry Up and Wait

Clicking ‘Publish’ on the dialog above begins the deployment process, which can easily take 15 minutes or more. During this time, some information is relayed through to Visual Studio about the actions being taken and their various statuses.

When the deployment is complete, click on the ‘Website URL’ link, or view the deployment in the Server Explorer and select ‘View in Browser’ from the right-click context menu.

The application finally appears. Note the URL is now that of an Azure cloud application.

In the Management Portal, expanding the Azure Subscription will show the Deployment node and all the compute instances fired up in the production environment. In our case, we have just one instance, but we could add more.

And there it is. A fully local web application that is now running in the cloud and using cloud-based blob and database storage.

Next we’ll talk about federating authentication to this application.

 

Migrating a Web Application to Azure: Step 3

Getting rid of file access

One issue to watch out for with an application in the cloud is file access.

File-Access in a highly-elastic environment such as Azure is generally a bad idea because it reduces the ability for a request to be correctly handled by multiple server instances. To combat this, generally, all access to the file system should be changed to access Azure storage, which does scale across multiple server instances.

In our example, we have contrived to have a picture stored in the database, but cached on the file-system on its first use, and subsequent access is to the cached file.

Local Application, Images from Server Files

We have code that looks like this in the original code:

public ServerPath EnsureImage(Picture picture)
{
    // ensure the directory exists
    var directoryPath = EnsureDirectory(RelativePathToImagesDirectory);

    // now ensure the file exists
    var filePath = EnsureFile(
        directoryPath,
        picture.OriginalFilename,
        _path =>
        {
            using (var stream = new FileStream(
                _path.Absolute,
                FileMode.OpenOrCreate,
                FileSystemRights.TakeOwnership | FileSystemRights.Write,
                FileShare.None,
                1024,
                FileOptions.None))
            {
                stream.Write(picture.Image.ToArray(), 0, picture.Image.Length);
            }
        });

    return filePath;
}

internal ServerPath EnsureFile(ServerPath root,
                               string filename,
                               Action<ServerPath> fileCreateFunc)
{
    var path = new ServerPath(Path.Combine(root.Relative, filename), Server.MapPath);

    if (!(File.Exists(path.Absolute)))
    {
        fileCreateFunc(path);
    }

    return path;
}

which results in

Note that the web application is still running locally, and the images refer to files located at relative paths within the server.

To remove the dependency on the file-system, we can use Azure Blob Storage to store the cached files.

The Rise Of The Blob…

Blob storage Azure is very cool. It presents a REST-based API to upload, tag, list and retrieve unstructured binary data like images, documents and videos.

So we can switch the application to interact with the Azure Blob store, referring to the images from the storage in the cloud.

To do this, we use the Azure storage account we created earlier.

The important pieces of information to capture are the Access Key (you get this when you click on the button with the orange outline), and the Azure storage account name which is highlighted.

We then initialize access to the blob storage account and container from within the application:

public void Initialize()
{
    if (CloudStorageAccount == null)
    {
        CloudStorageAccount = CloudStorageAccount.Parse("AccountName=cos204;AccountKey=<put_your_key_here>;DefaultEndpointsProtocol=http");
    }

    if (CloudBlobClient == null)
    {
        CloudBlobClient = new CloudBlobClient(CloudStorageAccount.BlobEndpoint, CloudStorageAccount.Credentials);
    }

    if (ThumbnailsContainer == null)
    {
        ThumbnailsContainer = CloudBlobClient.GetContainerReference("thumbnails");
    }

    ThumbnailsContainer.CreateIfNotExist();

    if (ThumbnailsContainer.GetPermissions().PublicAccess != BlobContainerPublicAccessType.Container)
    {
        ThumbnailsContainer.SetPermissions(new BlobContainerPermissions
                                               {PublicAccess = BlobContainerPublicAccessType.Container});
    }

    if (ImagesContainer == null)
    {
        ImagesContainer = CloudBlobClient.GetContainerReference("images");
    }

    ImagesContainer.CreateIfNotExist();

    if (ImagesContainer.GetPermissions().PublicAccess != BlobContainerPublicAccessType.Container)
    {
        ImagesContainer.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Container });
    }
}

What we are doing here is initializing the Blob Storage infrastructures and creating containers for Thumbnails and Images.

It’s important to note that the demo code shows a hardcoded Azure Connection String just to simplify the code snippet for reference. Normally this would be hidden away in a configuration file.

Local Application, Remote Images

We can now replace the file-access code shown above with its Azure Blob equivalent.

public ServerPath EnsureImage(Picture picture)
{
    var filePath = EnsureBlob(
        ImagesContainer,
        picture.OriginalFilename,
        _blob =>
            {
                var blobStream = _blob.OpenWrite();
                try
                {
                    blobStream.Write(picture.Image.ToArray(), 0, picture.Image.Length);
                }
                finally
                {
                    blobStream.Flush();
                    blobStream.Close();
                }
            });

    return filePath;
}

internal ServerPath EnsureBlob(CloudBlobContainer blobContainer,
                               string filename,
                               Action<CloudBlob> blobCreateFunc)
{
    var blob = blobContainer.GetBlobReference(filename);

    if (blobContainer.ListBlobs().FirstOrDefault(_blob => _blob.Uri.AbsoluteUri == blob.Uri.AbsoluteUri) == null)
    {
        blobCreateFunc(blob);
    }

    return new ServerPath(filename, blob.Uri.AbsoluteUri);
}

With this implementation of EnsureImage(), we get:

Pretty cool.

We now have a web application running on the local server, with data in the cloud, caching files and referencing Azure Blob storage also in the cloud!

Inspecting Blob Storage

While developing and debugging this stage of the migration, it’s useful to see exactly what is being stored in the Azure Blob storage.

The Windows Azure Management Portal does not have a handy tool to inspect the structure and contents of the Azure Storage account, so we turn to some other tools to do that.

The pack of Windows Azure Tools for Visual Studio 2010 v1.4 enables a storage account browser, which can be configured with the storage account and the key as before, allowing for full access to the storage account.

We can use this to ensure that the blob storage account is structured as we expect, and filled with the contents we expect.

Next, we’ll move the application itself to Azure.

Migrating a Web Application to Azure: Step 2

Migrating the database to SQL Azure

Now that we have a local web-application with a database, and a SQL Azure instance up and running, we can start the application migration by moving the database to SQL Azure.

SQL Azure is a modified version of SQL Server 2008 R2, and there are some small, but significant limitations on the SQL DDL commands that can be run on Azure. See here and here for more information.

The upshot is, if you already have a local SQL Server database, then you’re really much better off using some specialized migration tools to move it to Azure. The tool of choice for us is the SQL Azure Migration Wizard off codeplex.

Download and install the latest version of this tool on your development box. You’ll thank the tool-developer later!

Fire up the tool, and select “SQL Database” under “Analyze / Migrate”

Select the local database server which hosts the instance you want to migrate

Select the database(s) to migrate in turn.

In our case, we will have to migrate both the ASPNETDB and PICTURES databases.

Click through the dialogs, and migrate database object present in the database.

The Migration Wizard has created the script to use for the migration, and will now require your credentials to migrate to. This is where you fill in the details from the SQL Azure instance screen in the previous step.

Let’s let the migrator create a PICTURES database on the Azure Server

…and begin the migration work

…and we are done!

The Azure portal shows the two databases created and ready for use.

Now, let’s hook up the application we wrote to use these databases in the cloud. To do this, we point the connection strings in the Web.Config of the web application to the databases in the cloud.

Thusly:

And that’s it. Here’s the application running locally but with the data in the cloud.

I’ve created a new album called “Clouds” now, and uploaded three new pictures. If we look at the Azure database instance and look at the data in the PICTURES instance, we see the data there:

Next, we’ll talk about setting up the Azure workspace so we can move our application there…

Migrating a Web Application to Azure: Step 1

Setting up your Azure workspace

So we have a working web-application which we want to move to Azure.

The first thing to do is to get a Windows Live ID and line up one Windows Azure subscription and one SQL Azure subscription.

Sign into windows.azure.com with your Live ID

Click on the “Hosted Services, Storage Accounts and CDN” button on the bottom left panel, then on “Hosted Services” on the top left panel.

Right-Click on your Azure Subscription in the main panel and select “New Hosted Service”

Choose not to deploy at this stage!

Similarly create a Storage Account on the same subscription.

Now create a database server on the SQL Azure Subscription.

Ensure you set the firewall rules to include both your machine’s IP address and other Azure services. Your SQL Azure instance will need to be accessed from the Windows Azure instance,

Once you create the database server, you should make a note of the following pieces of information:

• The subscription id
• Your administrator login and password
• The instance name, and the fully-qualified database server DNS name

Now we have Windows Azure and SQL Azure instances set up for use.

Next we’ll migrate our database to SQL Azure.