Dynamic Robots.txt file for ASP.NET MVC Sites

Changing the contents of a robots.txt file when a site is moved from staging to a live environment is quite a manual and somewhat cumbersome process. I sometimes have the fear of forgetting to replace the "Disallow: /" line with correct indexable entries required for the website.

To give me one less thing to remember during my pre-live deployments, all my current and upcoming ASP.NET MVC sites will use a dynamic robots.txt file containing different entries depending on whether a site is in stage or live. In order to do this, we need to let the ASP.NET MVC application serve up the robots.txt file. This can be done by the following:

• Create a new controller called "SEOController"
• Add a new FileContentResult action called "Robots".
• Add a new route.
• Modify web.config.

SEOController

I have created a new controller that renders our "Robots" action as a plain text file. As you can see, my controller is not a type of "ActionResult" but a "FileContentResult". The great thing about "FileContentResult" is that it allows us to return bytes from the controller.

In this example, I am converting bytes from a string using Encoding.UTF8.GetBytes() method. Ideal for what we need to generate a robots.txt file.

[Route("robots.txt")]
[Route("Robots.txt")]
public FileContentResult Robots()
{
    StringBuilder robotsEntries = new StringBuilder();
    robotsEntries.AppendLine("User-agent: *");

    //If the website is in debug mode, then set the robots.txt file to not index the site.
    if (System.Web.HttpContext.Current.IsDebuggingEnabled)
    {
        robotsEntries.AppendLine("Disallow: /");
    }
    else
    {
        robotsEntries.AppendLine("Disallow: /Error");
        robotsEntries.AppendLine("Disallow: /resources");
        robotsEntries.AppendLine("Sitemap: http://www.surinderbhomra.com/sitemap.xml");
    }

    return File(Encoding.UTF8.GetBytes(robotsEntries.ToString()), "text/plain");
}

RouteConfig

Since I add my routing at controller level, I add the "MapMvcAttributeRoutes" method to the RouteConfig.cs file. But if you prefer to add your routes directly here, then this method can be removed.

public class RouteConfig
{
    public static void RegisterRoutes(RouteCollection routes)
    {
        routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

        routes.MapMvcAttributeRoutes(); //Add this line!

        routes.MapRoute(
            name: "Default",
            url: "{controller}/{action}/{id}",
            defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
        );
    }
}

Web.config

Add a "runAllManagedModulesForAllRequests" attribute to the modules tag in the web.config file to allow our robot.txt text file to be rendered by ASP.NET.

<system.webserver>
  <modules runallmanagedmodulesforallrequests="true"></modules>
</system.webserver>