One of the things I always forgot to add to my web applications is the Robots.txt file that Search Engines use to see what they should index. This file and sitemaps help make your site easier to navigate by the bots and allow them to understand what's legal and what you'd rather not have the revealed in their engines. I usually add any administrative pages or account pages even though they're protected by security, no need for the login page to be index if they sniff the link.

So how do you add Robots.txt to your MVC application? Glad you asked, here is a little code to get you started.

Sample Code
1.  Select the controller you would like to use for the robots.txt output.  I chose the HomeController in my application as I use it for most “top level” generic links like about us, contact us, index, etc.

2. Now, create a method called Robots to handle the request.
#region -- Robots() Method --
public ActionResult Robots()
{
    Response.ContentType = "text/plain";
    return View();
}
#endregion

3. Add the Robots.cshtml view to your Controller’s View directory.  Here is the code I have in my view, yours will vary.
@{
    Layout = null;
}
# robots.txt for @this.Request.Url.Host
User-agent: *
Disallow: /Administration/
Disallow: /Account/

4. Load up the class you are using to control your routes, if you are in an Area, this could your AreaRegistration class.  If you are at the top like I am and using the standard MVC template, this is probably the Global.asax.cs file.  Add your route to this file, mine looks like this.

routes.MapRoute("Robots.txt",

                "robots.txt",

                new { controller = "Home", action = "Robots" });

5. Compile and test.

If you've got an internet facing site, the chances are you will have a bot find you're request this page. you might as well give them the benefit of the doubt and allow them to understand where you want them to go. additionally you may save yourself some error log when this page is requested and no controller is found.

Just like something in ASP.NET, there are many ways to solve this riddle, if you use a different approach, please feel free to share it in the comments.

HostForLIFE.eu ASP.NET MVC 6 Hosting
HostForLIFE.eu is European Windows Hosting Provider which focuses on Windows Platform only. We deliver on-demand hosting solutions including Shared hosting, Reseller Hosting, Cloud Hosting, Dedicated Servers, and IT as a Service for companies of all sizes. We have customers from around the globe, spread across every continent. We serve the hosting needs of the business and professional, government and nonprofit, entertainment and personal use market segments.

In this blog post, I'll discuss a useful yet easy to implement document viewer API. Using that you can display a source file (e.g. Word, Presentation, Diagram, PSD) from your server to your browser without downloading it.
 
Some Use-Cases
    Display a Word resume to a user in a web browser
    Render a Visio Diagram on the web page
    View a Presentation or a Slide online without downloading it

Implementation
Now as the use-cases are clear, we will dive into the API implementation. It'll be an ASP.NET MVC application. We'll pull data/documents from the database and save it to a stream. You have to specify the file format in order to render it correctly. The source document will be then rendered to HTML. Eventually, our controller will return this stream to the View/browser.  
    public ActionResult Index()  
    {  
         License lic = new License();  
         lic.SetLicense(@"D:/GD Licenses/Conholdate.Total.NET.lic");  
         MemoryStream outputStream = new MemoryStream();  
         //specify just the file name if you are pulling the data from database   
         string fileName = "sample.pdf";  
      
         FileType fileType = FileType.FromExtension(Path.GetExtension(fileName));  
      
         using (Viewer viewer = new Viewer(() => GetSourceFileStream(fileName), () => new LoadOptions(fileType)))  
         {  
               HtmlViewOptions Options = HtmlViewOptions.ForEmbeddedResources(  
                    (pageNumber) => outputStream,  
                    (pageNumber, pageStream) => { });  
               viewer.View(Options);  
         }
         outputStream.Position = 0;  
         return File(outputStream, "text/html");  
                   
    }  
    private Stream GetSourceFileStream(string fileName) =>  
                new MemoryStream(GetSourceFileBytesFromDb(fileName));  
      
    //TODO: If you want to pull the data from the DB  
    private byte[] GetSourceFileBytesFromDb(string fileName) =>  
                System.IO.File.ReadAllBytes(fileName);  

HostForLIFE.eu ASP.NET MVC 6 Hosting
HostForLIFE.eu is European Windows Hosting Provider which focuses on Windows Platform only. We deliver on-demand hosting solutions including Shared hosting, Reseller Hosting, Cloud Hosting, Dedicated Servers, and IT as a Service for companies of all sizes. We have customers from around the globe, spread across every continent. We serve the hosting needs of the business and professional, government and nonprofit, entertainment and personal use market segments.