WARNING!: This is an advanced feature. You must consult Google's Webmaster Tools or some other outside source before you make any edits to your robots.txt file. Improperly editing this page can cause damage to your SEO.
These resources explain the functions and limitations of this file:
About /robots.txt (The Web Robots Page)
Learn about robots.txt files (Google Help)
The robots.txt file is used by search engines to categorize and archive web sites. It can be used in conjunction with Google's Webmaster Tools to optimize changes to your site. For example, if you have an existing site and are relaunching as a Metro Publisher website, you can use alterations to the robots.txt file to inform Google of the various changes from the old site to the new one. This can help minimize the adverse temporary effects caused whenever relaunching a website.
For more information about how specifically to use your robots.txt file including the exact lines of code you should input, you will need to consult Google's Webmaster Tools support documents.
Example of a robots.txt file:
The image above shows the default robots.txt file for Metro Publisher sites. These lines of code direct search engines to the sitemaps for your website. It is an essential part of how your site is indexed.
User-agent: * Disallow: Sitemap: http://design.metropublisher.net/sitemap.xml Sitemap: http://design.metropublisher.net/sitemap_news.xml
Please be aware that Google does not guarantee that disallowing access will prevent indexing:
Search engines can only index pages that they know about, so blocking the page from being crawled usually means its content won’t be indexed. While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the future.
Source: Google Blog - A note on unsupported rules in robots.txt
If you would like to alter or customize the robots.txt page, use Pages to do so.
- Log in as an editor and click on "Pages" from the main navigation.
- Then at the bottom of the page select "Add" from the menu.
- On the subsequent screen, select "Text."
- Add the title.
NOTE: This will not appear within the content of the page, so you may call it anything you want.
- Name the new page simply "robots.txt"
NOTE: This is the only name that you can give it. If you name it any other way, the search engines will not find it.
- DO NOT assign it to a section or subsection.
- Add whatever code you need.
NOTE: Unless you are an advanced user, you MUST consult webmaster tools to edit this page. If you edit this page improperly, search engines will not index your site properly.
- Remember to "Save."
That's it. Wait a few minutes for your new page to clear the cache.