Sometimes it can be beneficial to hold back some of your web pages from the search engine’s index, by deindexing them, thereby preventing the search engine spiders from rating them. 

Website owners tend to focus solely on getting their pages indexed. So much so they don’t even think about those pages that might drag their SEO efforts down – pages that need to be deindexed. 

Some people will even go so far as to try and avoid any of their pages getting deindexed, for fear of harming their ranking. 

But here’s the thing – not every page on your website needs to be indexed. Some of the pages on your website could even harm your SEO efforts, rather than help them. 

Doorway pages, for example, might seem like a good idea to help steer search engines in the right direction around your site, but they are in fact considered a black hat SEO tactic, and the use of them can result in you being penalised in the SERP. 

So, actually, getting pages that are harming your SEO efforts deindexed could help drive more traffic to your site, than if you were to leave them indexed. 


Essentially, whilst it is true that if you publish a lot of content on a lot of different pages that include the keywords and all the variants of the keywords that you want to rank for, this will help you to rank for those keywords. It can also be beneficial to hold back some of those pages from the search engine’s index, by deindexing them, thereby preventing the search engine spiders from rating them. 

What does indexing and crawling your site mean?

So we’ve explained crawl depth and frequency in a different SEO guide, but essentially, in order for your site and its specific pages to be indexed by the search engines and therefore be presented as results to users, the search engines have to know what your site is all about. 

They do this by sending out ‘spiders’ to crawl your site; to follow the pathways that you have connected through internal links, and by crawling through all your pages and checking out the content on each page, validating your HTML code and making sure the hyperlinks are valid and not spammy. 

The search engines then use all of the information that the spiders found on your site to determine where you fit in the relevancy to searchers’ requests. 

Indexing is what happens when your pages are added to a search engine’s index – i.e the bank of pages that the search engines search when trying to find relevant results for end users. 

If your web pages are indexed, then the Google spiders will crawl them and ascertain where you sit in the SERP. 

If your pages are deindexed and therefore not included in the search engine index, then your page won’t be considered and won’t be presented to end users. 

So you might be wondering why on earth then you would create a page that Google doesn’t take into account. Well, to put things a little more into perspective – WordPress pages and posts are by default, automatically indexed. So if you’re building a website on WordPress, every page you create is going to be indexed, even the little filler pages that look cute, but serve no discernable purpose to the end user. 

And whilst it’s good to have exposure for your pages, some of them just don’t need to be indexed. Because (and here’s the clanger) you can over optimise your website and ruin your site’s chances to rank highly.

Which is why deindexing can help.

Why deindexing can boost your SEO efforts

One of the key reasons that you might want to deindex a page is if it contains duplicate content. Duplicate content is when you have several different versions of the same webpage on your website, all with the aim of boosting your authority. And whilst you might be the go-to expert in your field, if you have lots of pages all saying the same thing, and Google can’t determine the differences between them, then you’re going to be penalised. 

Or you might have created identical pages, only one is printer friendly and the other isn’t. That doesn’t mean they aren’t beneficial to the end user, they are, but as far as the search engines are concerned, they’re the exact same page. And they’re duplicate content. So you get penalised. 

Other pages that you might want to consider deindexing include: 

  • Your about you page (if it’s identical to your homepage). 
  • Thank you pages – the pages that you direct users to once they’ve completed an action such as downloading an e-book for example, or contacting you through your contact page. 

What noindex and nofollow tags mean

If you’re wondering what the difference is between noindex and nofollow:

  • Noindexing a page means the search engines can still crawl a page, but by tagging a page as noindex you are preventing the search engines from indexing that page and presenting that particular page as a search result. 
  • Nofollow is an instruction that you include on HTML code to prevent the search engines from following any of the links on the page. Doing this means you are stopping any link juice from being passed through a particular link to the site it leads to. Setting links as nofollow can prevent any misunderstanding if Google believes the links are being sold or somehow influencing another site’s ranking.