Prerender Application Level Middleware

In the previous post Use Prerender to improve AngularJS SEO, I have explained different solutions at 3 different levels to implement Prerender. In this post, I will explain how to implement a ASP.NET HttpModule as a application level middleware to implement prerender. Since we call it ASP.NET, so it is applicable for both ASP.NET WebForm and MVC.

Application Level Middleware Architecture

At first, let's review what's the appliaction level middleware solution architecture.

ASP.NET HttpModule - PrerenderHttpModule

From above diagram, the easest way to implement it in ASP.NET is to create a HttpModule. Here I named it as PrerenderHttpModule.

At first, let's create PrerenderHttpModule and register BeginRequest event.

#region Implement IHttpModule
/// <summary>
/// init
/// </summary>
/// <param name="context"></param>
public void Init(HttpApplication context)
{ 
    context.BeginRequest += context_BeginRequest;
}

/// <summary>
/// dispose
/// </summary>
public void Dispose()
{
}
#endregion

#region Begin Request
protected void context_BeginRequest(object sender, EventArgs e)
{
    try
    {
        Prerender(sender as HttpApplication);
    }
    catch (Exception exception)
    {               
        Debug.Write(exception.ToString());
    }
}
#endregion

In PrerenderHttpModule, the major method is Prerender(HttpApplication)

private void Prerender(HttpApplication application)
{
    var httpContext = application.Context;
    var request = httpContext.Request;
    var response = httpContext.Response;
    if (IsValidForPrerenderPage(request))
    {
        // generate URL
        var requestUrl = request.Url.AbsoluteUri;
        // if traffic is forwarded from https://, we convert http:// to https://.
        if (string.Equals(request.Headers[Constants.HttpHeader_XForwardedProto], Constants.HttpsProtocol, StringComparison.InvariantCultureIgnoreCase)
         && requestUrl.StartsWith(Constants.HttpProtocol, StringComparison.InvariantCultureIgnoreCase))
        {
            requestUrl = Constants.HttpsProtocol + requestUrl.Substring(Constants.HttpProtocol.Length);
        }
        var prerenderUrl = $"{Configuration.ServiceUrl.Trim('/')}/{requestUrl}";

        // create request
        var webRequest = (HttpWebRequest)WebRequest.Create(prerenderUrl);
        webRequest.Method = "GET";
        webRequest.UserAgent = request.UserAgent;
        webRequest.AllowAutoRedirect = false;
        webRequest.Headers.Add("Cache-Control", "no-cache");
        webRequest.ContentType = "text/html";

        // Proxy Information
        if (!string.IsNullOrEmpty(Configuration.ProxyUrl) && Configuration.ProxyPort > 0)
            webRequest.Proxy = new WebProxy(Configuration.ProxyUrl, Configuration.ProxyPort);

        // Add token
        if (!string.IsNullOrEmpty(Configuration.Token))
            webRequest.Headers.Add(Constants.HttpHeader_XPrerenderToken, Configuration.Token);

        var webResponse = default(HttpWebResponse);
        try
        {
            // Get the web response and read content etc. if successful
            webResponse = (HttpWebResponse)webRequest.GetResponse();
        }
        catch (WebException e)
        {
            // Handle response WebExceptions for invalid renders (404s, 504s etc.) - but we still want the content 
            webResponse = e.Response as HttpWebResponse;
        }

        // write response
        response.StatusCode = (int)webResponse.StatusCode;
        foreach (string key in webResponse.Headers.Keys)
        {
            response.Headers[key] = webResponse.Headers[key];
        }
        using (var reader = new StreamReader(webResponse.GetResponseStream(), DefaultEncoding))
        {
            response.Write(reader.ReadToEnd());
        }

        response.Flush();
        application.CompleteRequest();
    }
}

Also, in order to make the logic flexible and easy to configure, I have add a configuration class and IsValidForPrerenderPage() method

private bool IsValidForPrerenderPage(HttpRequest request)
{
    var userAgent = request.UserAgent;
    var url = request.Url;
    var rawUrl = request.RawUrl;
    var relativeUrl = request.AppRelativeCurrentExecutionFilePath;

    // check if follows google search engine suggestion
    if (request.QueryString.AllKeys.Any(a => a.Equals(Constants.EscapedFragment, StringComparison.InvariantCultureIgnoreCase)))
        return true;

    // check if has user agent
    if (string.IsNullOrEmpty(userAgent))
        return false;

    // check if it's crawler user agent.
    var crawlerUserAgentPattern = Configuration.CrawlerUserAgentPattern ?? Constants.CrawlerUserAgentPattern;
    if (string.IsNullOrEmpty(crawlerUserAgentPattern)
     || !Regex.IsMatch(userAgent, crawlerUserAgentPattern, RegexOptions.IgnorePatternWhitespace))
        return false;
    
    // check if the extenion matchs default extension
    if (Regex.IsMatch(relativeUrl, DefaultIgnoredExtensions, RegexOptions.IgnorePatternWhitespace))
        return false;

    if (!string.IsNullOrEmpty(Configuration.AdditionalExtensionPattern) && Regex.IsMatch(relativeUrl, Configuration.AdditionalExtensionPattern, RegexOptions.IgnorePatternWhitespace))
        return false;

    if (!string.IsNullOrEmpty(Configuration.BlackListPattern)
      && Regex.IsMatch(rawUrl, Configuration.BlackListPattern, RegexOptions.IgnorePatternWhitespace))
        return false;

    if (!string.IsNullOrEmpty(Configuration.WhiteListPattern)
      && Regex.IsMatch(rawUrl, Configuration.WhiteListPattern, RegexOptions.IgnorePatternWhitespace))
        return true;

    return false;

}

The priority of checking whether is valid for prerender page:

1. Constant -> EscapedFragment:  _escaped_fragment_

User Agent: Setting -> CrawlerUserAgentPattern

(google)|(bing)|(Slurp)|(DuckDuckBot)|(YandexBot)|(baiduspider)|(Sogou)|(Exabot)|(ia_archiver)|(facebot)|(facebook)|(twitterbot)|(rogerbot)|(linkedinbot)|(embedly)|(quora)|(pinterest)|(slackbot)|(redditbot)|(Applebot)|(WhatsApp)|(flipboard)|(tumblr)|(bitlybot)|(Discordbot)

1. Constant -> DefaultIgnoredExtensions

   \.vxml|js|css|less|png|jpg|jpeg|gif|pdf|doc|txt|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent

2. Setting -> AdditionalExtensionPattern
3. Setting -> BlackListPattern
4. Setting -> WhiteListPattern
5. At last, return false.

Register PrerenderHttpModule

Generally, we have two different ways to register a HttpModule in ASP.NET, 

1. Use HttpModule configuration in Web.config
2. Use DynamicModuleUtility (Microsoft.Web.Infrastructure.dll)

Here, I have added the logic to support both, and I have added a app setting to control which one we want to use. By default, it will use DynamicModuleUtility, as we don't need to configure HttpModule in web.config, it's automatical.

Note, in order to use this HttpModule, please configure your IIS Application Pool to integrated mode.

<!--If it's false, please configure http module for UsePrestartForPrenderModule-->
<add key="UsePrestartForPrenderModule" value="true"/>

1. Use DynamicModuleUtility (Microsoft.Web.Infrastructure.dll)

This is the default configuration. 

• Configre UsePrestartForPrerenderModule  to true or remove this setting

<!--If it's false, please configure http module for UsePrestartForPrenderModule-->
<add key="UsePrestartForPrenderModule" value="true"/>

• Add reference for Microsoft.Web.Infrastructure.dll
• Add a static class PrerenderPreApplicationStart for assembly prestart.  We need to use static method in static class.

public static class PrerenderPreApplicationStart
{
    public const string StartMethodName = "Prestart";
    static bool UsePrestart = !bool.FalseString.Equals(ConfigurationManager.AppSettings[Constants.AppSetting_UsePrestartForPrenderModule], StringComparison.InvariantCultureIgnoreCase);

    /// <summary>
    /// used to configure for PreApplicationStart.
    /// i.e. [assembly: PreApplicationStartMethod(typeof(PrerenderPreApplicationStart), "Start")]
    /// </summary>
    public static void Prestart()
    {
        if (UsePrestart)
        {
            DynamicModuleUtility.RegisterModule(typeof(PrerenderHttpModule));
        }
    }
}

• Register PreApplicationStartMethodAttribute in AssemblyInfo.cs

[assembly: PreApplicationStartMethodAttribute(typeof(PrerenderPreApplicationStart), PrerenderPreApplicationStart.StartMethodName)]

Once the ASP.NET application loads this assembly, it will trigger PrerenderPreApplicationStart.Prestart() method, then registers the PrerenderHttpModule.

2. Use HttpModule configuration in Web.config

This is a very common and easy way, what we need to do is to:

• Configure UsePrestartForPrerenderModule to false

<!--If it's false, please configure http module for UsePrestartForPrenderModule-->
<add key="UsePrestartForPrenderModule" value="false"/>

• Configure PrerenderHttpModule

<system.webServer>
  <validation validateIntegratedModeConfiguration="false"/>
  <modules runAllManagedModulesForAllRequests="false">
    <!--Configure PrerenderHttpModule when UsePrestartForPrenderModule is false; -->
    <add name="prerender" type="DotNetOpen.PrerenderModule.PrerenderHttpModule, DotNetOpen.PrerenderModule" />
    <remove name="FormsAuthentication"/>
  </modules>
</system.webServer>

Configuration Section in Web.config

I have added a configuration section PrerenderConfigurationSection for prerender options

• Declare the configuration section in web.config section group.

  <configSections>
    <!-- For more information on Entity Framework configuration, visit http://go.microsoft.com/fwlink/?LinkID=237468 -->
    <section name="prerender" type="DotNetOpen.PrerenderModule.Configuration.PrerenderConfigurationSection, DotNetOpen.PrerenderModule"/>
    <section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
  </configSections>

• Configure options.

  <!--prerender settings-->
  <!--CrawlerUserAgentPattern: "(google)|(bing)|(Slurp)|(DuckDuckBot)|(YandexBot)|(baiduspider)|(Sogou)|(Exabot)|(ia_archiver)|(facebot)|(facebook)|(twitterbot)|(rogerbot)|(linkedinbot)|(embedly)|(quora)|(pinterest)|(slackbot)|(redditbot)|(Applebot)|(WhatsApp)|(flipboard)|(tumblr)|(bitlybot)|(Discordbot)"-->
  <!--WhiteListPattern, BlackListPattern: will check raw URL, which includes query string-->
  <!--AdditionalExtensionPattern: will only check extension-->
  <prerender ServiceUrl="http://localhost:3000" 
             Token="" 
             WhiteListPattern="" 
             BlackListPattern="" 
             AdditionalExtensionPattern="" 
             ProxyUrl="" 
             ProxyPort="80"/>

You can go to my github wiki page to get more details about each option: Configuration & Check Priority  

Nuget Package

I have created a nuget package, which is very convenient if you don't want to dive deep into the source code. 

• Install Nuget Package in your project.

           Visual Studio -> Tools -> Nuget Package Manager -> Package Manager Console.

Install-Package DotNetOpen.PrerenderModule

           If you want to take a look more detail about this package, you can go https://www.nuget.org/packages/DotNetOpen.PrerenderModule/ 

           There are several versions

1. Version 1.01-1.0.2, they are for .NET Framework 4.6.2
2. From Version 1.0.2, I have changed the .NET Framework version from 4.6.2 to 4.0, so that more people can use it. Which means the latest Nuget Package is using .NET Framework 4.0

• Register PrerenderHttpModule and configure options for prerender service.

            I have fully documented how to do this in my github wiki page, you can go there take a look.

1. Prerender Middleware for ASP.NET

2. Configuration & Check Priority            

• Done, try it out.

Github Project

I also have created a github project to host all source code includes sample code for testing: https://github.com/dingyuliang/prerender-dotnet, in this project, it includes ASP.NET HttpModule, ASP.NET Core Middleware, IIS Configuration 3 different solution. 

For ASP.NET HttpModule, you can go to https://github.com/dingyuliang/prerender-dotnet/tree/master/src/DotNetPrerender

Prerender Related

1. Use Prerender to improve AngularJS SEO
2. Setup Prerender Service for JavaScript SEO
3. Prerender Implementation Best Practice
4. Prerender Application Level Middleware - ASP.NET HttpModule
5. Prerender Application Level Middleware - ASP.NET Core Middleware

------------------------------------------------------------------------------------------------