Website Crawl Permissions | What Search Engines Can Index

Crawl permissions are used in order to prevent many issues affiliated with website indexing. When search engines index a website, they will by default crawl everything that can be reached from internal site references. Setting crawl permissions allows you to manage how your website is indexed. Crawl permissions is a top level category over robots.txt and internal meta data.

You can instruct search engines to crawl, ignore or read but not to index. Lets see how that can become beneficial,

I want search engines to follow a page but not to index it?

This is commonly used on websites with lots of dynamic search results. You can instruct search engines to follow the links on a page but not to index it, this is simply done through meta data, using a noindex, follow attribute. This is simply done by adding a meta instruction to the top of the document, learn more about this on the meta tags page.

I want search engines to completely ignore a specific page?

This is another common scenario for various types of pages, this can be done through a robots.txt file by setting the URL as a disallow, or by adding a meta attribute to the top of the document that instructs search engines to nofollow and noindex a page.

For more information visit the meta tags page or the robots.txt page.

I want search engines to index a page but not to follow the links on the page?

This is another common scenario for webmasters, this is easily done by adding a meta attribute to the page, adding a index, nofollow to the meta attributes will allow search engines to index the page, but not to follow any of the links on the page.

For more information visit the meta tags page.

These are just some of the permissions that can be set by various external website factors. For most SEO projects, these will be common scenarios in order to create a sound SEO structure on the website itself.

Crawl Permissions

So how do search engines actually crawl?

Well, the term crawl is applied as search engines follow paths laid out by the design of the website. As the website is constructed, the links built internally on the pages will be responsible for how the website is crawled.

Setting the permissions will help maintain a good profile, by preventing the indexing of potentially duplicate documents, by reducing page rank spread and much more.