Search engines and 500 internal server error

If your pages are not indexed by search engines, try to simulate their user agents. Make sure that the page can be viewed in that way and you do not get any 500 internal server error.

Possible causes:
- Creating SEO friendly urls at runtime using ASP.NET 2.0

Solution:
The default browser files are located in "C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\CONFIG\Browsers".
You need to add the browser definition for the new user agent to either this directory or or to App_Browsers directory under the application path.
http://msdn2.microsoft.com/en-us/library/ms228122.aspx

Browser definition files must have a .browser file name extension.

You may want to check this list to test if your application is working with other search engines' user agents.
http://www.spidermatic.com/en/supported_spiders.php

To simulate the user agents of search engines;
- Install User agent switcher addin to Firefox.
https://addons.mozilla.org/en-US/firefox/addon/59
- Create a new user agent in the options section of the add in.
- Set User Agent to the search engine's user agent.
For Google: "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)".
For Yahoo : "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
- Just put dummy text for the description. Let's say "Google Bot"
- When you run the same page on the browser with the new user agent selected, you should be able to see the page, otherwise search engines will not be able to crawl your page

You can also use Fiddle on IE to customize the user agent.
- Select Rules->Customize Rules menu option
- Go to the end of OnBeforeRequest method
- Add this line
oSession.oRequest["User-Agent"] = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
-Save the changes

You can see the user agents having problem from IIS web logs under system32\LogFiles directory