A couple months after migrating to subversion, we took another significant step to improve our build process by setting up a continuous integration server using CruiseControl.net and NANT. As explained in the previous post in this blog series, our new SVN repository structure supported a clear separation of development, staging and production environments. Now we needed something to assist in making sure that commits and changes to the repository resulted in actual builds being generated on the appropriate server.

Prior to this time, this was always a manual process where files were often dragged and dropped via windows explorer for simple deployments and for more complex deployments, services were stopped, sql scripts were executed and multiple applications were deployed in precisely timed order. As you can imagine, this was error prone, time consuming, stressful and sometimes not easily repeatable or easy to roll back.

Enter Cruise Control and NANT. Now, code promotions and deployments are nearly effortless and self documenting. One of my goals was to ensure that our CI (continuous integration) system made code promotions as simple as possible. I knew that the more difficult or cumbersome the system was, the less likely that my team would use it correctly or use it all.

This post is intended to offer a guided tour of our CI internals. I will go into some detail on our ccnet configuration and NANT scripting, but this is not intended to be a tutorial on the setup of cruise control and NANT scripting. It illustrates how my team has chosen to use these technologies, but every environment is different and I encourage you to dig into the documentation and other blogs to discover the full range of possibilities these tools provide.

In short, here is the flow of our CI system:

All Visual Studio projects and solutions involved in an application are registered with CruiseControl via config file entries in ccnet.config.
As code is committed to the trunk, ccnet automatically builds the project that contains the committed code. If there are errors, the dev team is alerted.
When a project is successfully built, its owning solution(s) will be rebuilt and the build files will be deployed to a development server.
When an application release is promoted to staging and the trunk is copied to the staging branch for that application, CruiseControl will build the solution in the SVN staging branch and deploy the build files to the staging servers.
When a release is ready for production launch, an assigned team member will go to the CruiseControl dash board and “Force” the production build. This will rebuild the solution from the staging branch in SVN and deploy the build files to the production servers. It will then copy the staging branch to a production tag for that release.

Registering Projects and Solutions with CruiseControl

In order for Cruise Control to be aware of our Visual Studio projects and solutions and therefore build and deploy them, they must be registered in Cruise Control’s configuration file ccnet.config.

Originally we added all configuration settings (or <project>s) to this file directly. However, as we added multiple ccnet projects, this config file became quite large and unmanageable. To bring some order to this, we use XML entities to define external files that contain configuration for a particular application or team. This external file is treated like an “include” file. So if you are adding projects for a new application or team, create a new entity in the ccnet.config file. This is a two step process:

1. Define the entity within the <!DOCTYPE> of ccnet.config. Here is an example:

<!DOCTYPE cruisecontrol [
    <!ENTITY app1 SYSTEM "file: app1.xml">
    <!ENTITY app2 SYSTEM "file:app2.xml">
    <!ENTITY app3 SYSTEM "file:app3.xml">
]>

The entity defines a name and then points to a file which should be located in the same directory as ccnet.config.

2. Reference the entity name in ccnet.config where you want the content of the entity file to be included. All of the entities are referenced immediately after the opening <cruisecontrol> element:

    &app1;
    &app2;
    &app3;

After setting up the entity, you begin to add projects to the entity file. A “project” is an XML element in the ccnet.config that defines a branch in SVN that CruiseControl should watch for changes and then take action if changes are detected. There are several kinds of actions that CruiseControl can take. Typically, these actions include executing an NANT script, possibly running unit tests and sending an email.

We create a separate project element for each of the following:

Every project in the VS solution.
The VS Solution on trunk to be deployed to dev servers
The VS Solution on the staging branch to be deployed to staging servers
The VS Solution on staging to be deployed to production servers

CCNet Project Naming Conventions

We will dive into the details of a Cruise Control project in a moment, but let’s first cover project naming conventions. Every project is given a name and this is the same name that is displayed in the Cruise Control dashboard and desktop tray. Projects are listed in both of those areas in alphabetical order. The convention we use is as follows:

An Example name is: APP1_STAGE_SOLN_admin. This is part of the App1 team, from the SVN staging branch, it’s a solution and it’s the admin solution.

This makes looking at the cruise control dashboard much more user friendly especially when you have lots of projects like we do. Furthermore, cruise control allows you to group projects into logical “categories.” As you will see when we look at a project config later, cruise control allows you to give each project configuration a category label. You can then filter your projects in the Cruise Control dashboard easily using these categories.

CCNet Project Naming Conventions

An Example name is: APP1_STAGE_SOLN_admin. This is part of the App1 team, from the SVN staging branch, it’s a solution and it’s the admin solution.

Anatomy of <project>

Here is a typical Project configuration in ccnet.config:

<project name="App1_TRUNK_PROJ_AdminPanel" queue="trunk” >
    <workingDirectory>D:\BuildSys\CCNetProjects\Libraries\AdminPanel.Core</workingDirectory>
    <artifactDirectory>D:\BuildSys\CCNetProjects\Libraries\AdminPanel.Core\artifacts</artifactDirectory>
    <category>App1</category>
    <webURL>http://devweb01.com:8082</webURL>
    <modificationDelaySeconds>15</modificationDelaySeconds>
    <triggers>
        <intervalTrigger name="continuous" seconds="30" buildCondition="IfModificationExists"/>
    </triggers>
    <sourcecontrol type="svn">
        <trunkUrl>http://svnsrv01. com/repos/search/trunk/library/AdminPanel</trunkUrl>
        <workingDirectory>..\..\trunk\library\AdminPanel</workingDirectory>
        <executable>C:\Program Files (x86)\CollabNet Subversion Server\svn.exe</executable>
        <autoGetSource>false</autoGetSource>
        <tagOnSuccess>false</tagOnSuccess>
    </sourcecontrol>
    <tasks>
        <nant>
            <executable>D:\BuildSys\Executables\nant-0.86-beta1\bin\nant.exe</executable>
            <buildFile>D:\BuildSys\CCNetProjects\genericlibrary.build</buildFile>
            <buildArgs>-debug- -D:Project=AdminPanel -D:solutionorprojectfilename=AdminPanel.csproj -D:solutionlist=AdminPanel -D:branch=trunk -D:type=library -D:stagebase=</buildArgs>
            <targetList>
                    <target>TriggerSolutionBuild</target>
            </targetList>
        </nant>
    </tasks>
    <cb:include href="ccnetpublisherschange.xml"/>
</project>

Consult the CruiseControl documentation if you are interested in a detailed explanations of each element. This config tells ccnet to check SVN at http://svnsrv01.com/repos/search/trunk/library/AdminPanel every 30 seconds and to execute the genericlibrary.build NANT script if it detects any change in the repository.

Almost all of our <project> configs look like this. The configs that build VS solutions call a different NANT script and I’ll explain that in detail later. One element that should be removed from the config for production deployments is the <triggers> tag. Production project configs should just have an empty <trigger/> element. This is because, at least in our environment, you never want an automated process to kick off a deployment to live servers. We want to launch these builds manually. I realize that the word “manual” has certain negative connotations, but in this case it means clicking on the cruise control “Force” button. I don’t think that’s too much to ask.

CCNet Project Naming Conventions

An Example name is: APP1_STAGE_SOLN_admin. This is part of the App1 team, from the SVN staging branch, it’s a solution and it’s the admin solution.

Anatomy of <project>
Here is a typical Project configuration in ccnet.config:

Building Visual Studio Projects and Solutions

The way we structured the repository monitoring strategy of our CI system was to have separate cruise control projects that watch for changes in a single Visual Studio project. If the project build is successful, then the project build script (genericlibrary.build) creates a dummy text file and adds it to a special SVN branch for each VS solution that depends on it. In our environment, a Visual Studio solution is essentially an application. We have another set of cruise control projects that monitor the above SVN branches and build the entire app (VS solution) when they see a modification to the branch which is triggered by the addition of the dummy text files.

A closer look at NANT build script genericlibrary.build

NANT is a powerful build scripting framework based on ANT. It provides functionality for source control and file system operations along with many other features. It has all of the flow control capabilities supported by most programming languages along with the ability to declare, inspect and manipulate variables. In our environment, the NANT script is the work horse of the build process.

Our genericlibrary.build script handles all commits on individual VS Projects. It simply checks out all source code in the project, builds it and then creates the dummy text file discussed above which triggers a solution build.

Here is our genericlibrary.build script:
<project>
        <description>${Project} build</description>
    
        <property name="debug" value="true" overwrite="false" />
    
    <property name="nant.settings.currentframework" value="net-3.5" />
    
    <property name="projectpath" value="${branch}\${stagebase}\${type}\${Project}\${solutionorprojectfilename}" />

NANT scripts contain one to many <target> nodes. Each of these represents a distinct unit of work. They may also depend on other target nodes. If a target node depends on another, the target specified in the depends attribute is executed first. When the ccnet.config <project> executes a NANT script, the config can specify a TargetList of targets to be called in specific order.

The solution NANT script

Here is the script that builds the solution.

    <project>
    <description>${foldername} build</description>
    <property name="debug" value="true" overwrite="false" />
    <property name="nant.settings.currentframework" value="net-3.5" />
    <property name="publish.dir" value="Apps\${Project}\artifacts\buildarchive\${branch}\${CCNetLabel}" />
    <property name="publish.dir.current" value="Apps\${Project}\artifacts\buildarchive\${branch}\${foldername}" />
    <property name="projectfile" value="${branch}\${stagebase}\${solutionorprojectfilename}" />
    <property name="executablepath" value="${branch}\${stagebase}\${type}\${foldername}${execsuffix}" />
    <property name="projectpath" value="${branch}\${stagebase}\${type}\${foldername}" />
    <property name="deploy_to" value="" unless="${property::exists('deploy_to')}" />
    <if test="${type=='web'}">
        <property name="stage_config_file" value="${branch}\${stagebase}\${type}\${foldername}\Web${branch}.config" />
    </if>
    <if test="${type=='services'}">
        <property name="stage_config_file" value="${branch}\${stagebase}\${type}\${foldername}\App${branch}.config" />
    </if>
    <target name="echo" >
        <echo message="${executablepath} solution at ${projectfile}" />
    </target>

This script pulls down the latest source code from SVN, builds it, then on each target server, the appropriate service is stopped, all binary output files and web content files are copied over, and the service is started. Finally, if this is a production deployment, a tag is created on the SVN production branch which serves as a snapshot of all source code at that moment in time.

Another important task which this script handles is making sure that the correct config file is copied to the deployment servers depending on the deployment environment. We will discuss this in more detail a little later.

The build script parameters

Both of our NANT scripts expect a number of input paramaters to tell it where the code exists in SVN, where it should be copied to, what service to restart, what solution builds to trigger, etc. These paramaters are specified in the <buildArgs> tag inside of the ccnet.config <project>. These parameters will be different depending on if you are creating a ccnet project for a VS project or solution. Here are examples:

VS Project:
<buildArgs>-debug- -D:Project=AdminPanel -D:solutionorprojectfilename=AdminPanel.csproj -D:solutionlist=AdminPanel -D:branch=trunk -D:type=library -D:stagebase=</buildArgs>

VS Solution:
<buildArgs>-debug- -D:serverpfx=devweb -D:execsuffix= -D:type=web -D:destinationpath=inetpub\wwwroot\adminpanel -D:solutionName=AdminPanel -D:solutionorprojectfilename=AdminPanel.sln -D:foldername=AdminPanel -D:servicename="W3SVC" -D:branch=trunk -D:serversuffixlist=01 -D:stagebase= -D:deploy_to=prod</buildArgs>

A VS Project build is much simpler than a Solution build. Project builds simply builds the project files and then creates a dummy file that gets committed to SVN for every dependent solution in the solutionlist param, which will trigger a solution build. A Solution build builds the entire solution, stops the servicename on every target server, copies the output files to the target servers and then starts the servicename. If the solution build is targeting production, it will take the additional step of creating a build tag in SVN.

The values in the buildArgs element parameters are name-value pairs:

–debug- turns debug off. If this were on all the time we would quickly run out of disk space due to output volume
–D:type is the name of the trunk parent folder that contains the project folder of the project in question. It will typically be “library” for basic class library projects or “web” for web applications.
–D:Project is the name of the project folder in the SVN trunk as trunk\${type}\${Project}
–D:solutionorprojectfilename is the name of the project file. This would be a .csproj file for projects and a .sln file for solutions. The file is relative to the folder given by the Project param for project builds. In our repository, all solution (.sln) files are located in the root of the trunk branch.
–D:solutionlist is a comma separated list of the solutions in which the project in question occurs. The solution name is actually the name of the folder that the solution build watches for changes. The project build will write a text file into this folder to trigger the solution build.
–D:branch is the name of the target branch to build, either “trunk” or “stage”
–D:stagebase is the name of the subfolder of the staging SVN branch that contains the application being built and deployed. It must be empty for trunk builds.
–D:serverpfx is the first part of the servername without the numeric suffix that code should be deployed to (ex. Devweb, web, etc).
–D:serversuffixlist is a comma separated list of suffixes for the servers to which deployment should be done. The first segment of server name is formed from serverpfx and a suffix, for example web and 01,02,03 will deploy to servers named web01, web02 and web03.
–D:destinationpath is the path on the server to which deployments will be done to receive the build files. For example, for a service this would be Services\<servicefoldername> or a web project might be inetpub\wwwroot\<webfoldername>.
–D:execsuffix is the path segment, relative to destinatioinpath, needed to access the file directory on the target servers that contain the build binaries. This is usually empty for web projects but “\bin\debug” for windows service projects.
–D:foldername is the name of the solution’s application project folder. This would be the project that hosts the application. Typically this would be the web or service project.
–D:servicename is W3SVC for a web project and the actual service name for a windows service project. During deployment this service is stopped and started
-D:deploy_to should be equal to “prod” for production deployment projects and otherwise empty.

An important note on app/web.config management

An application usually needs different configuration settings depending on its environment (development, staging, production). Our generic NANT scripts helps us manage this, but we must follow a convention in order for this to work.

In the <appsettings> element of the web/app.config file, we add this attribute: file=user.config. If a user.config file exists, any settings in that file will override those in the web/app.config.
A typical app will have up to 3 extra config files: user.config, userstage.config, usertrunk.config. user.config contains specific settings local to a developers personal machine, userstage is specific to the staging server(s) and usertrunk is specific to the shared development environment. The settings in the actual web/app.config are the production settings.
The generic NANT scripts will dynamically rename userstage.config to user.config on staging promotions and will rename usertrunk.config to user.config on dev. It does not deploy any of the user.configs on production promotions.

This convention ensures that the configuration settings unique to a specific environment can be kept under source control but that only the applicable settings are actually used in the appropriate environment.

What’s Missing?

Two things are embarrassingly deficient here:

Where are the nUnit tests in the NANT scripts? Yes, we have got to incorporate our nUnit tests into the CI system. Individual developers run the tests locally but that is not good enough. NANT has the capability to run nUnit tests and fail a build if the tests do not pass.
If you look at the parameters being passed into the NANT scripts, you will notice that the strings look very similar. We should be following the principal of convention over configuration here to make this much simpler to setup and easier to read. Rather than having several parameters with values of app1, app1.sln, d:/inetpub/wwwroot/app1, etc., we should collapse all of these parameters to a single parameter that takes an application name app1 and then embeds that into the other configuration strings.

Learning More

There is lots of good information out there on continuous integration generally and Cruise Control and NANT specifically. The docs for Cruise Control and NANT are a good starting point.

Cruise Control documentation: http://confluence.public.thoughtworks.org/display/CCNET/Documentation

NANT documentation: http://nant.sourceforge.net/release/latest/help/

While there is room for improvement, this system has worked great for us and has made our lives much easier. I can’t stress enough the importance of being able to deploy code with a press of a button. It took us a long time to make the implementation of this system a priority, but considering the time it saves us to work on feature development, we should have built it much earlier.

Hurry Up and Wait!

Tales from an automation engineer

The Perfect Build Part 3: Continuous Integration with CruiseControl.net and NANT for Visual Studio Projects /October 21, 2009 by Matt Wrock

Latest Posts