One of the items I continuously forgot to add to my web applications is the Robots.txt file that Search Engines use to see what they should index.  This file and site maps help make your site easier to navigate by the bots and allow them to know what's legal and what you would rather not have the published in their engines.  I usually add any administrative pages or account pages despite the fact that they're protected by security, no need for the login page to be index if they sniff the link.

So how do you add Robots.txt to your MVC three application?  Glad you asked, here may be a very little code to get you started.

Code

1. Choose the controller you'd wish to use for the robots.txt output.  I selected the HomeController in my application as i use  it for many “top level” generic links like about us, contact us, index, etc.

2. Create a method called Robots to handle the request.
#region -- Robots() Method –
public ActionResult Robots()
{
    Response.ContentType = "text/plain";
    return View();

}
#endregion

Add the Robots.cshtml view to your Controller’s View directory.  Here is the code I have in my view, yours will vary.
@{
    Layout = null;
}
# robots.txt for @this.Request.Url.Host 
User-agent: *
Disallow: /Administration/
Disallow: /Account/
Load up the class you are using to control your routes, if you are in an Area, this could your AreaRegistration class.  If you are at the top like I am and using the standard MVC template, this is probably the Global.asax.cs file.  Add your route to this file, mine looks like this.

routes.MapRoute("Robots.txt",               
"robots.txt",
new { controller = "Home", action = "Robots" });

Now, Compile and test.

If you have an internet facing site, the chances are you will have a bot find you're request this page. you might as well offer them the advantage of the doubt and allow them to know where you want them to travel. additionally you may save yourself some error log once this page is requested and no controller is found.

Just like something in ASP.NET, there are some ways to solve this riddle, if you employ a special approach, please feel free to share it within the comments.

HostForLIFE.eu ASP.NET MVC 6 Hosting
HostForLIFE.eu is European Windows Hosting Provider which focuses on Windows Platform only. We deliver on-demand hosting solutions including Shared hosting, Reseller Hosting, Cloud Hosting, Dedicated Servers, and IT as a Service for companies of all sizes. We have customers from around the globe, spread across every continent. We serve the hosting needs of the business and professional, government and nonprofit, entertainment and personal use market segments.

ASP.NET MVC 6 Web API Project

ASP.NET MVC 6 introduces several new project types after you initially pick that you want to develop an ASP.NET MVC 6 Web Application. One of those application types is the new Web API Project.


If you choose the Web API Project, a new Web API Controller Class is created for you to provide an example of responding to Get, Post, Put, and Delete requests for your API.

public class ValuesController : ApiController {

   // GET /api/values
    public IEnumerable<string> Get() {
        return new string[] { "value1", "value2" };
    }

    // GET /api/values/5
    public string Get(int id) {
        return "value";
    }

    // POST /api/values
    public void Post(string value) {}


    // PUT /api/values/5
    public void Put(int id, string value) {}

    // DELETE /api/values/5
    public void Delete(int id) {}
}

With the Web API Project you will also notice a new API specific route added to the RouteTable in Global.asax.cs.

routes.MapHttpRoute(
    name: "DefaultApi",
    routeTemplate: "api/{controller}/{id}",
    defaults: new { id = RouteParameter.Optional }
);

Running the project and navigating to ~/api/values will display a list of the values in XML Format. I removed the XML namespacing to keep things simple.

<ArrayOfString>
    <string>value1</string>
    <string>value2</string>
</ArrayOfString>

If you change the Accept Header so that you will only accept JSON, the same controller action will send the values via JSON instead.

["value1","value2"]

Web API Controller Class - ApiController in ASP.NET MVC 4

Creating a new Web API Controller Class is as simple as using the Add Controller Recipe in ASP.NET MVC 4 and choosing the Empty API controller Tempate.



Or, you could just create one via Add Item which has a new Web API Controller Class as an option.

I created a simple ProductsController that handles all the CRUD options for products in a mythical e-commerce website.

public class ProductsController : ApiController {
    private readonly IRepository<Product> _repository;

    public ProductsController(IRepository<Product> repository) {
        _repository = repository;
    }

    public IEnumerable<Product> Get() {
        return _repository.Queryable();
    }

    public Product Get(int id) {
        var product = _repository.Get(id);

        if (product == null)
            throw new HttpResponseException(HttpStatusCode.NotFound);

        return product;
    }

    public HttpResponseMessage<Product> Post(Product product) {
        _repository.Add(product);

        var response = new HttpResponseMessage<Product>
            (product, HttpStatusCode.Created);
        response.Headers.Location = new Uri(Request.RequestUri,
            Url.Route(null, new {id = product.Id}));

        return response;
    }

    public Product Put(int id, Product product) {
        var existingProduct = _repository.Get(id);

        if (existingProduct == null)
            throw new HttpResponseException(HttpStatusCode.NotFound);

        _repository.Save(product);

        return product;
    }

    public HttpResponseMessage Delete(int id) {
        _repository.Delete(id);

        return new HttpResponseMessage(HttpStatusCode.NoContent);
    }
}

You can see that in some instances I am just returning a Product and in other instances I am returning a more informational HttpResponseMessage. For example, in the case of the Post of a new Product, I need to tell the REST Client the new location of the newly added product in the header. In other actions I am also throwing a HttpResponseException if the resource requested is not found. Validation, Logging, and other concerns are being done in various ActionFilters just like in your normal ASP.NET MVC Projects. Try to pull those cross-cutting concerns out of the main logic as much as possible.

ASP.NET Web API OData Syntax for Paging and Querying

If you want to enable various paging and querying of products you can make a slight change to the Get ApiController Action and return an IQueryable<Product> as opposed to IEnumerable<Product>.

public IQueryable<Product> Get() {
    return _repository.Queryable();
}

Now from your browser you can add paging, filtering, sorting, and other options to shape the data. Here is an example call that does paging and sorting.

api/products?$skip=2&$top=2&$orderby=Title

The XML Response by the browser is:

<ArrayOfProduct>
    <Product>
        <Id>3</Id>
        <Title>RipStik</Title>
        <Price>69.00</Price>
    </Product>
    <Product>
        <Id>4</Id>
        <Title>Shred Sled</Title>
        <Price>49.00</Price>
    </Product>
</ArrayOfProduct>

Or the JSON Response:

[{"Id":3,"Price":69.00,"Title":"RipStik"},
{"Id":4,"Price":49.00,"Title":"Shred Sled"}]

Conclusion

ASP.NET Web API integration with ASP.NET MVC 4 is really slick. Now you can easily create an API for your website using the new ApiController Base Class to respond to REST Clients.