Robots Control
With the robots control functionality, tags can be implemented to prevent crawling of a document, below are common usage tags.
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
This version of the meta tag prevents Google and other search engines from Indexing or crawling the documents links. Applying both of these tags will prevent the page being indexed or acknowledged. This can be useful on pages that are either using duplicate content, or important login pages.
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
This version of the meta tag will prevent search engines from following the links on the page, but will allow search engines to index the page and its content. This is useful in situations for pages that should not inherit page rank. This is commonly found on pages that are not important towards rankings, such as privacy policy pages, summary pages, contact pages or other pages with a large volume of outbound links.
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
This version of the meta tag will prevent the search engines from indexing the page, but will allow the search engines to crawl links that are found on the page. This is useful and commonly implemented on search results pages which are dynamic, yet offer useful links to other parts of the website.
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
This version of the meta tag will allow search engines to crawl and index the page whilst crawling links on the page. Typically search engines will do this without this tag being present.
You should always seek professional advice before implementing tags to prevent crawling and indexing issues.
Meta Refresh Tags for Re Directs
Meta refresh tags have become obsolete but are sometimes used in areas where dynamic redirects are not possible. Meta refresh tags allow the document to re-direct the end user to another page through HTML code. This tag has become obsolete as it is classed as an unfriendly search engine redirect. This tag should not be used on important pages, and should be avoided at all possible costs. Pages with these re-directs may not be indexed correctly, or may loose ground in search engine positions.