Track Nuget Downloads using OData, RSS and Ifttt.com

In this post I am going to show you how you can be notified of new downloads of any Nuget package via email from a service that will poll Nuget every 15 minutes. If email sounds overly intrusive, there are other options. So If this sounds interesting, read on.

If you have open source projects hosted on Nuget and you are a bit on the OCD (obsessive compulsive disorder) side, you are frequently tempted to wander over to Nuget.org and check out your download stats. Well, I have finally started to notice that I spend a fair amount of time every day, checking the Nuget site as well as other sites that may provide key indicators of my projects’ health. For instance I like to check for new followers or twitter mentions. Unfortunately, a lot of this time is spent simply waiting for a web page to load and reveal to me that there is no new information. So not only is this unproductive but it can lead to negative thoughts and emotions.

There are many pharmaceutical options available here, but I am not a Medical doctor and it would be unwise for me to give advise of a psychiatric nature. However I do have some technical solutions that simply require a computer with access to the world wide web. If you lack either of these, I have nothing to offer and you should now leave this page.

Ok. good. It’s just you and me now….hmm…this is uncomfortably intimate. No matter…

Switch from a Pull to a Push model

What I found myself craving was a way to let all of this information come to me and announce to me that there is new data rather than me having to spend time pinging several sources for what is likely to be no new information. In my case, I really wanted my phone to beep or vibrate when I get a new download, follower or mention. For me, this would not be a nuisance given the small amount of data. If you owned jQuery, you may want a more unobtrusive notification. Fortunately the solution I am about to propose can channel notifications through a variety of mediums.

Enter If-this-then-that ifttt.com

A few months ago I noticed a new referring link on my blog from a domain called ifttt.com. I visited the link and perused the site and discovered that it provided a way of creating sort of mash ups of various social media. ifttt stands for If This Then That. And the site simply allows you to create rules of If something occurs (new tweet, RSS feed item, DropBox item, etc.) Then some other thing should be triggered such as an email sent or a tweet or facebook update, etc. I have to admit my initial impression was “That’s dumb.” Then about a week later Scott Hanselman blogged about this service having been duly impressed by its offerings. I still didn’t really get it.

Not sure why I didn’t see the value right away but I see it now. Last week I setup a number of tasks that have freed me of the constant compulsion to check these various web sites for new data. I have a rule that will send me an email whenever my project has a new Github follower or a new mention on twitter. I have tasks that tell me when I have new stack overflow comments or new stack overflow points. All of these tasks were relatively easy to set up using ifttt.com. ifttt’s very friendly GUI provides an extremely simple way to send an email to yourself triggered by a new tweet or RSS feed item.

Here is an example of the task that sends me an email when my project RequestReduce is mentioned on twitter:

It is honestly trivial to set this up.

But Nuget Has no RSS Feed with items representing downloads

Currently Nuget provides no RSS feed or any notification option for subscribing to download stats beyond what is displayed on the project search results and individual project details pages. I don’t know if there are plans to implement this by the Nuget team in the near future, but I wanted something up and running soon that didn’t need to be polished.

All Nuget data iavailable from the website is exposed through an OData feed

I knew that the data I was interested in was available via OData. There are a few posts out there that talk about this. I found that David Ebbo’s post had the detail I deeded to get started. With the name of any Nuget package Id, you can get its total download count via the public Nuget OData endpoint at http://packages.nuget.org/v1/FeedService.svc.

Here is an example query using LinqPad:

Creating a custom RSS Feed to broadcast new downloads

Currently as far as I can tell, there is no facility built into ifttt to consume this OData format. Yes, you can expose OData as an ATOM feed but given the Nuget schema, this would only be useful if you wanted to be notified of new versions. Essentially each version is a child entity of the master Packages entity. DownloadCount is simply a property associated with each version. Note that a version has both a VersionDownloadCount and a DownloadCount. The first is simply the count for a single version and the latter is the aggregate count of all combined versions released in a single package.

At first I tried playing with Yahoo Pipes and some other online RSS builder apps but none of these was going to work. At least not simply. I didn’t want to spend a lot of time on this since what I wanted was really quite simple and could be coded up fairly trivially. So I ended up just writing my own feed generator and I took the opportunity to create my first Azure application. I plan to blog more specifically on the azure specific details later and how they differed from my work with an AppHarhor application.

Here is the RSS Generator code:

public class FeedHandler : IHttpHandler{    private const string NugetServiceUri = "http://packages.nuget.org/v1/FeedService.svc";    private readonly IDictionary<string, IList<SyndicationItem>>         packageDownloadCounts = new ConcurrentDictionary<string, IList<SyndicationItem>>();

    public bool IsReusable    {        get { return true; }    }

    public void ProcessRequest(HttpContext context)    {        var packageName = context.Request.QueryString["packageId"];

        var nugetContext = new Nuget.GalleryFeedContext(new Uri(NugetServiceUri));        var last = (            from x in nugetContext.Packages             where x.Id == packageName && x.IsLatestVersion             select new { x.DownloadCount, x.Version }).First();

        var items = GetSyndicationItems(packageName, last.DownloadCount);        var nugetUrl = string.Format(            "{0}/Packages(Id='{1}',Version='{2}')", NugetServiceUri, packageName, last.Version);

        var feed = new SyndicationFeed("Nuget Download Count Feed",           "Provides the current total download count for a Nuget Package",           new Uri(nugetUrl), nugetUrl, items.Last().LastUpdatedTime,           items);        using (var xmlWriter = XmlWriter.Create(context.Response.OutputStream))        {            feed.SaveAsRss20(xmlWriter);            xmlWriter.Flush();            xmlWriter.Close();        }

        context.Response.ContentType = "text/xml";        context.Response.End();    }

    private IList<SyndicationItem> GetSyndicationItems(string packageName, int count)    {        IList<SyndicationItem> items;        lock (packageName)        {            if (packageDownloadCounts.ContainsKey(packageName))                items = packageDownloadCounts[packageName];            else            {                items = new List<SyndicationItem>();                packageDownloadCounts.Add(packageName, items);            }            var title = string.Format("{0} has {1} total downloads", packageName, count);

            if (!items.Any(x => x.Title.Text == title))                items.Add(new SyndicationItem(                   title,                   "",                   new Uri(string.Format("http://nuget.org/packages/{0}",                                         packageName)), Guid.NewGuid().ToString(),                   new DateTimeOffset(DateTime.UtcNow)));            while (items.Count > 20)                items.RemoveAt(0);        }

        return items;    }}

You can grab the full Visual Studio Solution from https://github.com/mwrock/NugetDownloadFeed. Not much happening here. Its just a handler that takes a packageId in the query string and then checks the odata feed to see if there are more downloads than there were since the last time it checked. If there are, it creates a new feed item.

ifttt.com will poll this feed every 15 minutes. I currently have this feed up and running at http://wrock.cloudapp.net/downloadFeed.axd. Anyone is free to use it but I provide no guarantee for stability or longevity. That said, I have no plan to change the endpoint or bring it down. However, I may clean the code up a bit and offer it as a Nuget package so that anyone can host their own feed.

Consuming the feed from an ifttt.com Recipe

Beyond the creation of “one off” tasks. ifttt provides a means of encapsulating common task logic into a reusable “Recipe.” These are handy if you find yourself creating the same task again and again with the only difference being a single variable. In my case here, I wanted to create three tasks. One for each of my Nuget projects. It also seemed reasonable that others may want to make use of this as well. So I created a recipe that anyone can use in order to create their own Nuget Download Notification task. Simply create an ifttt account (Super fast and easy to do) and go here: http://ifttt.com/recipes/9302.

As the directions state, simply replace my Package Id RequestReduce with the Package Id that you are interested in.

If you do not want to be notified by email, you have several different options. You could have it tweet from a specific account, send an SMS message or create an Evernote entry. And there are many more options than that.

I’d really like to hand it to the folks at @ifttt for creating this ingenious service and wish them the best of success!

Adding CSS Inheritance Resolution to RequestReduce Sprite Generation Process

UPDATE: This functionality was released on 1/9/12 in v1.7.0.

Currently RequestReduce is unable to sprite alot of background images because of its limitation of treating each css class as an atomic unit. Inheritable dimensions and padding is lost which either makes an image unspritable by RequestReduce or the sprite is malformed because of improperly calculated dimensions.

Further, pre-sprited images in a document about to be reduced are often malformed and rendered in a distorted fashion because it is common practice to place the background url in one widely inherited class and have more specific classes define just the background-position of the individual image in the sprite. This makes many initial renderings using RequestReduce give a bad first impression especially if the user does not read the wikis on preparing the sprites and spriting requirements here and here. It can also make the exercise of converting docs to be compatible with RR spriting a time consuming and tedious endeavor.

I had initially thought this was a small edge case but it is proving to be an all too common problem and I think the adoption story could be dramatically improved by adding this feature. This is the one feature that can break the RequestReduce golden rule: do not break the page I am reducing. The addition of this feature can prevent this.

Unfortunately it is not a small effort. However it is not monumental either.

Installing an HttpModule via Dropping in the bin without Editing Web.config

Last week another team wanted to use an HttpModule I had written. The problem is that they had no access to the hosting platform of their content. Their DLLs are consumed via MEF by a rather large infrastructure affectionately known as MTPS or what the world knows as the MSDN Library.

Unfortunately, the functionality provided by this DLL requires that it be run as an HttpModule. So I remembered reading a post of David Ebbo that described how to load an HttpModule dynamically at run time.The topic suddenly seemed relevant to this issue so I revisited his blog and applied his instructions to my assembly and wow…sure enough, just drop my dll in the bin and BAM! it loads and runs. If you would like to see a very small example app that does this please see my code sample on the MSDN Samples Gallery.

Technically, to accomplish this the code involved couldn’t be more trivial. Here is a quick walk through.

Specify a PreApplicationStartMethod

In the Assembly.info of the assembly containing the HttpModule, the below attribute must be included:

[assembly: PreApplicationStartMethod(typeof(Loader), "LoadModule")]

This line points to a Method inside the assembly which should be envoked during Pre Application Startup. This event occurs even before the Application Start event in the ASP.NET lifecycle. The official MSDN Library reference for this attribute can be found here. Of course this is assuming that there is a class either inside the assembly or referencable to the assembly of type Loader and that it contains a method called "LoadModule."

Dynamically Register the Module

Inside the LoadModule method, you will use the Microsoft.Web.Infrastructure assembly to actually register the module. The Microsoft.Web.Infrastructure assembly is not included in the .net BCL but you can easily download and install via Nuget. Here is a look at the entire Loader class:

using Microsoft.Web.Infrastructure.DynamicModuleHelper; 

namespace HttpModule {     public class Loader     {         public static void LoadModule()         {             DynamicModuleUtility.RegisterModule(typeof (DynamicModule));         }     } } 

As you can see there is not a whole lot to this. Just one line simply tells the run time the type name of the HttpModule to be registered. Here the HttpModule is inside a class called DynamicModule. The type that you pass to RegisterModule must derrive from IHttpModule.

Is this a good practice?

At first glance this seems really cool. It removes one setup step that consumers of the HttpModule would normally have to go through to install a third party module. It appears that this would provide a more friction free installation story for any third party module. What can be bad about that?

Well maybe I'm over thinking this but my first concern here is discoverability. What if the module has heavy hitting impact on the markup of the page. I envision a new developer or team inheriting such an application and wonder just how long it will take for them to find where this "alteration" is coming from especially in a sophisticated application with perhaps 20 referenced DLLs and lots of internal code. Or perhaps a team member drops in such a module and forgets to tell the team she put it there. I'm thinking that at some point in this story some negative energy will be exerted. Perhaps even a tear shed?

As an author myself of an open source HttpModule that can transform the markup of a site, this is of particular interest to me. My RequestReduce project is a module that combines and minifies CSS and Javascript and sprites and optimizes CSS background images on the fly. Since the default behavior of the module simply requires the module to be registered in the user's web.confg, if I were to employ the technique used in this sample, the module installation would have a truly plug and play installation story which sounds very awesome and I am tempted to add it. But I worry.

I'd be interested in others feedback in this matter. Do you think it is a good idea in general practice? Do you think the perhaps limited discoverability poses a true risk? In the end, do you think consumers would be more or less attracted to such an installation story?

Released RequestReduce 1.5 Supporting Custom Minifiers and a Critical Performance Fix for v1.4

I just released RequestReduce v1.5 on Nuget and on http://www.RequestReduce.com.

If you are currently on v1.4, you will definitely want to upgrade to this new version. There was a misconfiguration in v1.4 causing a Regex to recompile on every call. The impact will vary depending on the number of scripts and css links on your site but is could well be in the 100s of milliseconds.

Enough about that. The key feature that v1.5 adds is a small API allowing you to do the following:

• Log RequestReduce exceptions thrown from its processing thread to your own error logging implementation including very simple support for Elmah logging.
• Plug in a custom minifier to replace the RequestReduce defaulr Microsoft Ajax Minifier. You can also use this if you want to override the settings that RequestReduce uses.
• Filter out certain CSS, Javascript or Image resources from being processed by RequestReduce including entire pages or areas of your site from being processed. The API allows you to evaluate any property in the HttpRequest to decide if a resource should be filtered.

Here are the details on how to use the API:

Logging RequestReduce Errors

Especially if you already have a error logging mechanism, it is advisable that you log any errors encountered by the RequestReduce reduction processing. This will aid in bug reporting and generalk troubleshooting. To do this, you simply need to provide an Action delegate that will take charge of logging the passed Exception. Here is an example of how to use this if you use Elmah for error logging:

RequestReduce.Api.Registry.CaptureError(x => ErrorLog.GetDefault(null).Log(new Error(x)));

 

Injecting your own CSS or Javascript Minifier

RequestReduce attempts to follow a decoupled architecture which allows developers to swap out certain parts with their own behavior. To override RequestReduce's use of the Micosoft Ajax minifier library, you simply create a class that derrives from IMinifier. There is not much to IMinifier:

public interface IMinifier{    string Minify<T>(string unMinifiedContent) where T : IResourceType;}

Here Is RequestReduce's implementation:

public class Minifier : IMinifier{    private readonly Microsoft.Ajax.Utilities.Minifier minifier = new Microsoft.Ajax.Utilities.Minifier();    private readonly CodeSettings settings = new CodeSettings {EvalTreatment = EvalTreatment.MakeAllSafe};

    public string Minify<T>(string unMinifiedContent) where T : IResourceType    {        if (typeof(T) == typeof(CssResource))            return minifier.MinifyStyleSheet(unMinifiedContent);        if (typeof(T) == typeof(JavaScriptResource))            return minifier.MinifyJavaScript(unMinifiedContent, settings);

        throw new ArgumentException("Cannot Minify Resources of unknown type", "unMinifiedContent");    }}

It's not difficult to imagine how you would change this implementation to use something like the YUI Compressor for.Net. Lets say you had a YuiMinifier class that you want RequestReduce to use instead of its own minifier. It might look something like this:

public class YuiMinifier : IMinifier{    public string Minify<T>(string unMinifiedContent) where T : IResourceType    {        if (typeof(T) == typeof(CssResource))            return CssCompressor.Compress(unMinifiedContent);        if (typeof(T) == typeof(JavaScriptResource))            return JavaScriptCompressor.Compress(unMinifiedContent);

        throw new ArgumentException("Cannot Minify Resources of unknown type", "unMinifiedContent");    }}

You would just need to add the following code to your startup code:

RequestReduce.Api.Registry.RegisterMinifier<YuiMinifier>();

That's it. Now your minification code will be used.

Filtering Resources from the Reduction Processing

If you would like to have certain CSS or javascript resources filtered from the reduction processing or if you would like to exclude certain images or even entire requests from all response transformations, then use the AddFilter method:

public static void AddFilter(IFilter filter)

 

RequestReduce provides four concrete types deriving from IFilter:

 

public class CssFilter : Filter<CssJsFilterContext>public class JavascriptFilter : Filter<CssJsFilterContext>public class SpriteFilter : Filter<SpriteFilterContext>public class PageFilter : Filter<PageFilterContext>

All of these derrive from:

public class Filter<T> : IFilter where T : class, IFilterContext

The IFilter implementations really don't do anything on their own other than pass a IFilterContext which exposes various properties of the request. Each of the IFilter implementations provide a single constructor which takes a Predicate where T is an IFilterContext that RequestReduce populates and provides to the predicate delegate.

There are three different implementations of IFilterContext available:

public class CssJsFilterContext : IFilterContext{    public HttpRequestBase HttpRequest { private set; get; }    public string FilteredUrl { private set; get; } //The Css or JS url to be processed    public string FilteredTag { private set; get; } //The entire HTML tag being processed}

public class PageFilterContext : IFilterContext{    public HttpRequestBase HttpRequest { private set; get; }}

public class SpriteFilterContext : IFilterContext{    //BackgroundImageClass has several properties that include all the CSS background attributes    public BackgroundImageClass BackgroundImage { private set; get; } }

Depending on the type of filter (Page, css, javascript or sprite) for each item that RequestReduce processes, it will provide the appropriate IFilterContext to your predicate. If you return true, RequestReduce will eliminate that resource from being processed. Here is an example of how to have RequestReduce ignore any image over 100px X 100px.

RequestReduce.Api.Registry.AddFilter(new SpriteFilter(x => x.Width > 100 && x.Height > 100));

 

Make sure to add these filters inside your App start or some other pre application, one time execution block.

Microsoft’s MSDN and Technet Forums and Search adopt RequestReduce for Web Performance Optimization

This week RequestReduce was launched on Microsoft’s MSDN and Technet web sites specifically with the Forums and Search applications. So if you ever land on a Forums page within these sites or conduct a search within the MSDN or Technet environment, RequestReduce is crunching and merging the CSS and javascript as well as spriting their background images resulting in about a 25-30% Reduction in the number of HTTP requests made.. RequestReduce already services the MSDN and Technet gallery sites which include Code Samples and the Visual Studio Gallery. All of these sites combined represent several million page views a day and prove that RequestReduce can perform under considerable load.

This migration process did surface some “edge case” bugs in the Sql Server cache flushing logic that would likely only surface in high traffic environments. The Sql Server cache is an add on assembly that is aimed to synchronize the RequestReduce generated Scripts, CSS and images across multiple servers and ensure that users do not view different or broken content when moving from one server to another but rather guarantees a consistent user experience throughout a session.

The discovery of these bugs provided the bulk of the code that went into this week’s 1.4 Release of RequestReduce. The release was largely a fix and polish release but here are some highlights:

Dashboard Page

Troubeshooting issues in the Forums/Search migration inspired me to create a dashboard page to assist in catching and isolating some tough to catch bugs. The page is currently extremely bare bones and rather cryptic. It displays what css and javascript has been processed, what is queued for processing and what is currently in the middle of being processed. It also provides links for flushing individual merged css/js files from the cache. The page can be accessed from:

http://<your site>/<RequestReduce virtual directory>/dashboard

Its greatest benefit is to assist in determining what RequestReduce is currently up to if anything and where RequestReduce is in terms of processing all queued content. Expect this page to get much more informative and maybe prettier in coming releases.

Performance Optimization

The javascript processing introduced in release version 1.3, added a lot of work to the Response Filter which has to iterate over every character of the response stream to the browser. This is the single piece of code that needs the most perf optimization since it is the only request blocking activity within RequestReduce. Its speed has a direct impact on the Time to First Byte download of the hosting RequestReduce enabled web page.

So I changed some List<T> implementations to arrays and removed some unnecessary lines of code. Granted these are micro optimizations but even 50 – 100ms of time filtering a page is far too long and this process has to be extremely optimized. Its also a bear to read, write and change which I plan to address in a future release. Sure I could eliminate several lines of code and overly complicated logic by using regular expressions, but that would actually degrade performance significantly.

Speaking of regular expressions. I do make extensive use of them in the background processing of content. Especially in CSS parsing for sprites and imports. So I eliminated a lot of my static Regex instances and replaced them with instance member Regex’s in a singleton class. .net will cache the most recently used 15 static regular expressions in an application and I do not want to infringe upon that cache especially when I have about 8.Using a singleton allows me to manage my own cache.

Support for CSS @Media

CSS media types is something that can be applied to an entire CSS stylesheet, instructing the browser to use its styles only for specific media. For example: print and screen are the most popular and obvious. RequestReduce had been oblivious to the Media attribute of the Link tag and @import directive. This caused some CSS to break on certain sites. Especially if the last CSS on a page was targeted for Print media and strips out a lot of styling. This was resulting in the web experience having the styles striped out as well. No longer. RequestReduce now wraps the contents of any <Link/> or @import inside @media braces to preserve the media constraints they specify.

Support for ScriptResource.axd scripts and all other Compressed content

RequestReduce already supported the minification and merging of WebResource.axd files. However, RequestReduce always assumed that the content it was requesting was uncompressed text since it did not specify compression in its Accept-Encoding request headers. Well it ends up that ScriptResource.axd files violate HTTP by sending GZIP’d content regardless of the value in this header. This is a show stopper for an upcoming large RequestReduce adoption in the near future that includes pages with over 30 to 40 CSS and script files including tons of ScriptResource.axd files. RequestReduce now looks to see if a Response is compressed with gzip or deflate and uncompresses accordingly. RequestReduce leaves the recompression up to IIS.

This really makes RequestReduce an attractive solution for sites mired in asp.net 2.0/asp.net ajax constructs that had good intentions of encapsulating scripts but horrible perf implications.