The Perfect Build Part 1

A year ago, my team was using Visual Source Safe as our version control repository and our builds were built and deployed manually using sophisticated tools like windows explorer and remote desktop. This was not a happy time.

• VSS was awfully slow and practically impossible to work with remotely.
• VSS branching, tagging and merging capabilities were primitive at best.
• Our manual deployments were not scripted and therefore not easily repeatable and certainly not guaranteeably repeatable.
• Deployments were inadequately documented and it was often impossible to know exactly what was deployed and when.
• Complicated deployments were time consuming, error prone and just plain stressful for the build engineer.

Well all of this has changed now. We migrated to Subversion for source control and implemented CruiseControl and NANT as an integration server and deployment solution. Looking back on this migration, I can honestly say that it represents the biggest leap forward we have made in terms of process improvement. This is a first in a series of blog posts where I will document this process in detail. I will outline how we tackled the following:

• Repository structure: We did not want to merely copy our VSS repository tree. We wanted to create something fundamentlly more useful.
• Generic NANT scripts to handle Visual Studio Projects and Solutions
• Managing configuratioin files in different environments (dev, staging and production)
• Automated tagging production deployments
• Cruisecontrol organizaion

Some may say, "Its only source control, what's the big deal?" Well our source is our product. Any inability to produce our product or any risk to the integrity of our product is a big deal. As developers, we interact with source control throughout every working day and it is imperative that these interactions are simple, reliable and valuable. If there is waste in terms of time or lost code at this layer of the develoopment process, its like a death by a thousand cuts to the whole dev team.

This was a migration that I took very seriously. Like most software shops, we are always very busy and juggle lots of projects at the same time. This migration provided no direct revenue increase to our business and was always difficult to prioritize. I was often asked "why don't we just migrate now? Just have Joe Jr. Developer port the code from VSS to SVN and be done with it." This was something I really wanted to think through, research and do right. I knew it was not going to be a "set and forget" operation. It needed to be planned, implemented and enforced and baby sat. My team, including myself, had been using VSS for years and were not familiar with the branch/merge paradigm of SVN. I knew we needed to understand these concepts and use them to their fullest. I also knewthat developers were going to experience hickups with usig SVN and its related tools. I needed to educate myself and make myself availale as a mentor of this technology although I was a novice. There was no budget to bring in a consultant.

So I researched on the web, I talked to colleagues with experience in SVN in varying environments, and I mapped out my own custom solution. My tech department is composed of different teams that produce and maintain several different applications that are web based, desktop and background services. We also have some common libraries shared and maintained by all teams and we share third party tools and libraries some of which are open source and require constant updates to new versions. We needed a solution that would allow us to build separate applications maintained by separate teams with code shared accross the organization. We needed a setup that would allow developers to commit code to a centralized Trunk that had all of the most recent changes but would present these changes to different applications in a controlled manner.

I beieve we have acieved exactly this and more. So what do we have now?

• We have a source control repository that informs us of exactly what is on an individual dev's environment, on our integrated dev box, on staging and on production.
• We have a setup that allows us to see exactly what code was deployed on which application on which date.
• Commits to SVN are automaticaly built and deployed to our integration dev server
• Commits to staging are automaically built and deployed to our staging servers.
• With a click of a button in CruiseControl, an application can be deployed to our production server farms
• Each environment can have its own versioned configuration
• Rolling back to a previous deployment is trivial

My next post in this series will focus on the SVN structure we developed and how it partitions our different environments and applications but ensures that each new build incorporates the latest comon code.

Debugging Windows Services in Visual Studio

One challenge involved in developing windows service applications is debugging them in Visual Studio. I don't know why Visual Studio does not provide better support for this, but I've seen some creative techniques employed to make debugging windows services possible. One popular method is to put a really long thread.sleep(30000) in the OnStart event and then install the service, start it and attach the debugger to the service's process hoping that it will take less than 30 seconds to start it, find the process and attach.

There is a better way and it is quite trivial.

There is one prerequisite: Make sure you do not have service logic in your OnStart() method. This turns out to be a good thing either way. I've seen 100 line service OnStart methods that put a good deal if not all of the logic into this one method. Well what if you want to reuse this logic in a console, WPF or win forms app? Its not a very flexible methodology. I believe that a windows service project should never contain more than your installer and entry point class and the entry point class only handles start, stop and pause by caling into a completely separate assembly. However, before over philosophizing this, lets quickly jump into how to setup your service project so that it can be easily dubugged in VS.

Follow these steps:

1. Change your Main() entry point method to Main(string[] args)
2. Change the Output Type property of the servce project from Windows Aplication to Console Application
3. Your Main(string[] args) method shouldlook something like:

        static void Main(string[] args)        {            if (args.Length > 0)            {                MyApp app = new MyApp();                app.Run();            }            else            {                ServiceBase[] ServicesToRun;                ServicesToRun = new ServiceBase[]                 {                     new Service1()                 };                ServiceBase.Run(ServicesToRun);            }        }

Finaly, in the Debug section of the project properties, provide a command line argument.

Your OnStart() should contain the same app startup code as you Main method:

        protected override void OnStart(string[] args)

        {

            MyApp app = new MyApp();

            app.Run();

        }

Thats it. Now hit F5 and you will see a command window pop up and all of your break points should be recognized. MyApp contains the meat of te logic. Main and OnStart are just dumb harnesses.

Debugging Managed Production Applications with WinDbg

Yesterday our issue tracking software was hanging and the vendor was not responding to our ticket requsts (They are nine hours ahead of us). The application is a .NET application so I decided to capture a crash dump and dive in with windbg. I have a love/hate relationship with windbg. I love it because it provides a vast wealth of informationvirtually telling me EVERYTHING thats going on with my process. It has saved my behind several times. I hate it because I don't use it frequently enough to have all of the cryptic commands memorized and often have to relearn and reresearch the commands I need to use in order to solve my problem. Windbg is not for the faint of heart. There is no drag and drop here. But if you have an app bugging out on a production server and don't want to attach a debuger to it, windbg is the tool for you.

This post is an adaptation of a document I created for my team and me a few years back. I use it like a cheat sheet to help me get started quickly.

When servers start crashing and/or hanging in production, often the only recourse you have is to capture a memory dump of the ailing process and analyze it using Microsoft’s Native debugger – WinDbg. Without this tool, you may just be shooting in the dark. These techniques can not only be applied to web applications but to any application – under managed or unmanaged code.

A memory dump will allow you to see everything going on in the captured process: executing threads and how long each have been running, stack traces of all threads and even the values of parameters passed to functions. They can also be used to troubleshoot memory leaks, allowing you to see what is in the heap.

A word of caution is in order: windbg is a pain to use. At least that has been my experience. There is almost no documentation included and the commands are very unintuitive, and this is compounded by the fact that you (hopefully) rarely use it.

There are three basic steps to this process:

1. Preparing the debugging environment on the problem server.
2. Actually capturing the dump while your process is crashing or crashed.
3. Analyzing the dump in windbg

Preparing the Debugging Environment

There are a few steps to complete to get the server ready:

1. Install Microsoft’s Debugging toolkit. Get the latest version at http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx. Note that there is a 32 bit and 64 bit version. If you are running on a 64 bit server but you have a managed app that is compiled for 32 bit, you will need to use the 32 bit version of windbg to debug.
2. Create an environment variable for path to symbol file (.pdb files that contain information that map native instructions tp function calls). Create a system environment variable called _NT_SYMBOL_PATH with the value: C:\symbols\debugginglabs*http://msdl.microsoft.com/download/symbols;C:\symbols\debugginglabs;C:\Program Files\Microsoft.Net\FrameworkSDK\symbols;C:\windows\system32
3. Copy sos.dll from the Microsoft.net directory to the same directory where you installed the debugging toolkit. This file provides extensions to windbg for analyzing managed code.

Capturing a Memory Dump

This step can be a bit tricky depending on the circumstances of your crashing behavior. There are typically 3 ways to do this:

1. Call up a test URL to see if the app has crashed or is locking
2. Use Task Manager to see if the CPU is pinned
3. Use Performance Monitor and look for queueing threads. If threads are queueing, that means that all available .net worker threads are busy which usually means something is wrong.

Once you have determined that the process has crashed, bring up a command prompt and navigate to the directory where you downloaded the debugging toolkit. Next type:

adplus.vbs –hang –pid [process ID of problem process]

If there are more than one worker process running and you are not sure which one is causing problems, repeat the above command for both processes.

This command will launch windbg in a separate window to load the process information. Just let it run and it will close when it completes.

Analyzing the Dump

1. Open windbg.exe which is inside the directory that you extracted the debugging toolkit to.
2. Go to File/Open Crash Dump and find the dump (.DMP) file you just captured. It will be in a subfolder of the debugging toolkit directory.
3. type .load sos.dll to load the managed code extensions.

You are now ready to start troubleshooting. Below are some commands I commonly use to get useful information. At the end of this document are some links to some MS white papers with more detailed information on performance debugging.

Listing all threads and how long they have been running

!runaway

Note the thread IDs of any particularly long running threads. If you have several threads that have been running for minutes, that could point to a never ending loop that is eating CPU or just a long running background thread.

Listing Managed Threads

!threads

There are several noteworthy tidbits here:

Lock Count: If this is greater than 0, it means that the thread is waiting(blocking) for another thread. For instance it might be waiting for a DB query to come back or a response from a socket. If you have a bunch of these, it could be a tip that there is a bad query. See below on how to get the call stack of an individual thread to see exactly what it is doing.

Domain: This is the address of the app domain that the thread is running in. This is very helpful if you have several web sites running in the same worker process. Once you find the problem thread(s), you can use this to see which web app is causing the problem. Keep in mind that all asp.net workerprocess have a “default” app domain used for launching new app domains (there is one per web app) and handling GC.

Determine which Web Application a thread is running in

!dumpdomain [address]

This dumps a list of assemblies loaded into the domain which should tip you off as to which web app it is running in.

Get a summary information on the Threadpool

!threadpool

This tells you haw many threads are free/in use and what the CPU utilization was at the time of the capture.

Get the stack trace of a single thread including passed parameters

~[thread id]e !clrstack –p

Get the thread ID from !threads or use “*” to get a dump of ALL threads.

Get detailed information on an object

!dumpobj [address]

This gives info on all fields in the object.

More Resources

http://msdn.microsoft.com/en-us/library/ms954594.aspx
This is an old link but has good and thorough informatioin.

http://blogs.msdn.com/tess/ This is Tess Ferrandez's blog. She has tons of great posts on this subject and also on analyzing memory leaking problems.

Web Site Performance - It's not just your code

I'm a software developer. By default, I tend to focus on optimizing my code and architecture patterns in order to tune performance. However, there are factors that lie well outside of your application code and structure that can have a profound impact on web site performance. Two of these factors that this post will focus on are the use of a CDN and IIS compresion settings.

Over the last month, me and my team have contracted a CDN and tweaked compression settings resulting in a 40% improvement of aveage page load time outside of our immediate geographical region!

Using a CDN for static content

A CDN (content delivery network) is a company that has data centers distributed over the globe. Some of the dominant players are Akamai, CDNetworks and Limelight. These have many data centers dispersed over the United States as well as several scattered internationaly. Chances are they have servers that are located much closer to your users than your own servers. You may not think this is a big deal. What's a couple hundred milliseconds to load an image? Well if your site has alot of images as well as css and javascript (jquery alone packs a hefty download size), the boost can be  significant. Not only will average load time improve, but you will be less suseptible to general internet peaks and jitters that can potentially cause significant and annoying load delays.

One of the disadvantages of using a CDN is cost. You will need to work with a commercial CDN if you want to have your own images and other static content that you have authored to be hosted via CDN. However, several popular frameworks are freely hosted on the CDNs of large internet companies. For example just this week Scott Guthrie announced that Microsoft will be hosting jQuery and the asp.net AJAX libraries. You can also use Google's CDN to serve jQuery. I use Yahoo's YUI ajax libraries and they can be served from Yahoo. Another cost related consideration is that the bandwidth you would be paying to serve this content yourself is now absorbed by the CDN.

One other disadvantage that I often hear related to CDNs is that the control and availability of the content is now at the mercy of the CDN. Well I personally feel much more comfortable using Google's vast network resources than my own.

Migrating to a CDN for the hosting of your own media is fairly transparent and painless especially if you already host all of this content on its own domain or sub domain (ie images.coolsite.com). The CDN will assign you a DNS CNAME and you will add that to your domain's zone file. Now all requests for static content will go to the CDN's servers. If they do not have the content, they will go to your server to get it. Then all subsequent requests will go to the CDN's cache. You can specify what the cache expiration will be and you should also be able to manually flush the cache if you need to.

One other perk that most commercial CDNs provide is detailed reporting on bandwidth and geographical data telling where your content is being requested from.

Our servers are based in California's Silicon Valley and we realized a 30% performance boost in the midwest and eastern United States. It should also be noted that our pages are very image light. So a site that has lots of rich media has even more to gain.

Compress Everything

I had always been compressing static content but not dynamic content. I think I was scared off by the warnings that the IIS documentation gives regarding high CPU usage. Yes, compression does incur CPU overhead, but with today's CPU specs, chances are that it is not significant enough to keep you from turning on this feature. Our servers tend to run at about 5 to 10%. After turning on dynamic content compression, I saw no noticable CPU increase but I did see a 10% increase in page load performance. All of this is free and took me just a few minutes to configure. In fact it is better than free, you will save on bandwidth.

I did do some poking around the web for best practices and found that it is worth while to tweak the default compression levels in IIS. Here is a good blog article that goes into detail on this topic. To turn on static and dynamic compression at the ideal levels on IIS 7. I issued these commands:

C:\Windows\System32\Inetsrv\Appcmd.exe set config -section:urlCompression -doStaticCompression:true -doDynamicCompression:true 

C:\Windows\System32\Inetsrv\Appcmd.exe set config
-section:httpCompression -[name='gzip'].staticCompressionLevel:9
-[name='gzip'].dynamicCompressionLevel:4

Tools for analyzing page load performance

Here are a few nice tools I use to observe and analyze web site performance:

1. Firebug. A must have Firefox plugin that will tell how long each resource takes to load.
2. An external monitoring service. I use Gomez. This will not only tell you how long it takes for your site to load, but it can monitor from around the globe and provide very rich and detailed reporting. I have several alerts configured that page me if my site is taking too long to load or is broken. "Broken" can mean 500 errors, Server too busy errors, non responsive servers due to server outages or bad DNS or it can even mean a failed pattern match of strings expected on the page.
3. YSlow. This is a firefox plugin from Yahoo that works with Firebug and analyzes several key indicators on your site. It examines your headers, caching, javascript, style sheet usage and much more and then gives you a detailed list of possible improvements you can make.

So if you feel that you have done all the code tweaking you can to get the most performance from your site, think again and take a look at these tools to see how the outside world is experiencing your content.

Colocatng ASP.NET MVC and Web Forms in the same Web Application

My team and I are migrating a large web forms/ADO web application over to the MVC framework using nHibernate and Domain Driven Design principles. After dabbling in MVC and nHibernate (although I have used MVC in java in a past life) and reading Eric Evans' book, I've been chomping at the bit to implement these frameworks in our less than clean production code.

As I mentioned above, this is a large application and I am not about to suspend our feature backlog while we spend months converting web forms to controller/views. Instead I have decided to keep the current collection of web forms pages that we have now and "fold in" conntrollers and views as we build out new functionality. Any minor enhancements to existing web forms will remain in the current web form structure, but any changes requiring heavy lifting will warrant a conversion to MVC. I should note that our "model" is thankfully already abstracted into separate class libraries.

In order to accomplish this happy coexistence of web forms and MVC, we have done the following:

1. Created a brand new MVC project for the views.
2. Created a new project for controllers to keep the controllers and views separate (we are using the Sharp Architecture Framework)
3. Added a "WebForms" directory to MVC Views projects
4. Copied the web forms tree from our legacy web application project to the new WebForms folder
5. Made minor changes to relative URL references

This has worked out really well for us. It has allowed us to keep a large and long lived application functioning as is while providing a framework for future functionality that is a much better enforcer of separation of concerns and far more unit test friendly.

We are taking a similar migration strategy with our Model layer. We have sevral class libraries of domain classes riddled with ADO code (yuck). Again, we are continuing to use these classes and extend them with small changes. However, large changes and new domain objects are following a DDD approach using nHibernate. There has been a definite learning curve with nHibernate and Fluent nHibernate, but I can't tell you how much cleaner the code looks without all of the database plumbing. You see only what you need to see -- your business logic not to mention we are no longer having to create countless stored procedures for the simple CRUD functionaity.

I don't think this architecture is suitable for every application. It would most likely be a poor choice for our external web application that serves over ten million page views a day. But it is ideal for a large administrative CRUD application supporting a complex domain model -- a perfect candidate for DDD.