What is web page indexing?

How are web pages found?

Web pages are found by being indexed. Indexing is the process of search engines visiting and analysing new and updated web pages before adding them to their index (database) of webpages. Indexing can be done in three ways:

• Let the crawlers do the work

• Submit a sitemap (in XML format)

• Request indexing manually

Content Management Systems are often configured to allow indexing of newly published pages by default. But what if a landing page, for instance, that’s only to be accessible to a select group of recipients isn't required to be indexed by search engines? i.e. to not be visible in search engine results pages (SERPs).

If your website developer added a content managed Robots Directive to a page so content editors can control how a page is indexed, then it's simply a case of setting the page to NOINDEX/NOFOLLOW. This means the page on your site will not be indexed and its in-page links not followed by search engines.

If however, there isn't a Robots Directive field available to content editors then assigning the directive may require you to talk to your development team, who'll update the robots.txt file.

Why indexed web pages are important

The indexing of URLs is essential for them to be found by search engines and subsequently found by potential customers interested in your product or service.

If web pages aren’t assigned to be indexed they won't be found, and consequently won't rank.

A URL's ranking and Domain Authority isn't static. As well as being dependent on continual optimisation, there are also other factors at play. Such as:

• How often content is published

• How authoritative and relevant content is

• Internal linking

• External linking (backlinking)

• Competing pages

• User behaviour when a link in a SERP is browsed 

The work of search engines is constant. Website indexation never stops. Crawlers periodically return to websites to check for updates and assess their ranking. That’s why it’s important to ensure your web pages are created to achieve the best ranking results.

At Codehouse, all our website builds include a Robots Directive which allows content editors to assign the following:

• INDEX / FOLLOW:  Index the page/Follow its links

• INDEX / NOFOLLOW: Index the page/Don’t follow its links

• NOINDEX / FOLLOW: Don’t index the page/Follow its links

• NOINDEX / NOFOLLOW: Don’t index the page/Don’t follow its links

Working with Codehouse

Our knowledgeable digital experience team and Google certified experts can help your business get the very best from its content marketing strategy. Get in touch to find out more.