{"id":10174,"date":"2019-10-11T12:00:55","date_gmt":"2019-10-11T12:00:55","guid":{"rendered":"https:\/\/www.improvemysearchranking.com\/?p=10174"},"modified":"2024-10-31T11:28:13","modified_gmt":"2024-10-31T11:28:13","slug":"optimise-website-crawlability","status":"publish","type":"post","link":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/","title":{"rendered":"How to Optimise Website Crawlability"},"content":{"rendered":"

As you may know, Google crawlers crawl websites regularly to see if there is any content that they could index. While this is usually an automated process, there are a few things that you can do to optimise it and make it easier for Google crawlers to find the most important content on your website and rank it in the search engine result pages (SERPs).\u00a0<\/span><\/p>\n

 <\/p>\n

<\/p>\n

A finite amount of time<\/b><\/h2>\n

Before we discuss how you can optimise website crawlability and make it easier for Google to index your web pages, it is important to understand one important concept.<\/span><\/p>\n

Whenever Googlebot visits a website, it has a finite amount of time for discovering and crawling web pages and links. Once that period of time expires, the Googlebot stops crawling.\u00a0<\/span><\/p>\n

The time for a revisit depends on multiple factors, but that\u2019s not our topic of discussion. What is important to understand is the fact that if you do not clear the path for Googlebot, it won\u2019t be able to reach many of your web pages.<\/span><\/p>\n

Ultimately, it may lead to a lower indexed page count and organic traffic, and it will be your loss.<\/span><\/p>\n

So, here are a few ways you can optimise website crawlability and make it easier for Google to index your website and discover more pages.<\/span><\/p>\n

 <\/p>\n

Determining the index ratio<\/b><\/h2>\n

Before you proceed any further, it is important to determine the index ratio for your website.\u00a0<\/span><\/p>\n

This ratio shows the number of pages that Google is indexing in comparison to the total indexable pages that are on your website.<\/span><\/p>\n

This figure not only tells you how many pages Google was actually able to find, but it also indicates the pages that Google actually deemed important enough to index.<\/span><\/p>\n

 <\/p>\n

Analysing non-indexable pages<\/b><\/h2>\n

Because Googlebot has a limited amount of time to crawl a website, it should not be crawling non-indexable web pages.\u00a0<\/span><\/p>\n

Therefore, you should check how many of these non-indexable pages are being crawled. You should also analyse whether it makes sense to make some of those pages available for indexing.<\/span><\/p>\n

 <\/p>\n

Disallowed URLs<\/b><\/h2>\n

Almost all websites have a few URLs that search engine crawlers do not have access to. While it is completely fine to block Googlebot\u2019s access to certain pages, it is essential that you review them from time to time.<\/span><\/p>\n

Some of those web pages might be important for indexing or for helping Googlebot discover other pages for crawling. In that case, make sure you are not blocking Googlebot from accessing them.<\/span><\/p>\n

 <\/p>\n

Removing 404 error pages<\/b><\/h2>\n

Error pages, such as those with 404 errors, are of no use to anyone. If Googlebot crawls such pages, it is only wasting valuable time. Therefore, you should fix 404 pages every time you can.<\/span><\/p>\n

Moreover, it is also important to remember that Google crawlers will keep visiting 404 error pages periodically to see if they are live again. Therefore, the best way is to remove 404-error web pages completely to optimise the crawlability of your website.<\/span><\/p>\n

 <\/p>\n

Conserve crawl budget by minimising redirects<\/b><\/h2>\n

Each redirect in the redirect chain consumes Google\u2019s crawl budget and consumes valuable time because Google crawlers need to check every page in the chain.<\/span><\/p>\n

Therefore, it is recommended to minimise redirects and help Google crawl more efficiently.<\/span><\/p>\n

 <\/p>\n

Canonicalised pages<\/b><\/h2>\n

You might be aware of how canonicalisation works in SEO.<\/span><\/p>\n

If there is content duplication, the canonical tag helps identify search engines which web page to index and rank in the search engine result pages.<\/span><\/p>\n

However, to be able to decide which is the primary web page, Google still has to crawl all of them. This consumes and wastes Google\u2019s crawl time.<\/span><\/p>\n

Keep content duplication to a minimum level to help Google and optimise crawlability.<\/span><\/p>\n

 <\/p>\n

What\u2019s next?<\/b><\/h2>\n

By taking the aforementioned steps for your website, you can improve the technical health of your website, identify areas of crawl waste, and help improve Google crawl more efficiently and effectively.<\/span><\/p>\n

You can use\u00a0<\/span>Google Search Console<\/span><\/a>\u00a0and the free version of\u00a0<\/span>Screaming Frog<\/span><\/a>\u00a0to find the relevant information and learn more about the technical side of your website.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

As you may know, Google crawlers crawl websites regularly to see if there is any content that they could index. While this is usually an automated process, there are a few things that you can do to optimise it and make it easier for Google crawlers to find the most important content on your website […]<\/p>\n","protected":false},"author":8,"featured_media":18043,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_oct_exclude_from_cache":false,"inline_featured_image":false},"categories":[33,35],"tags":[],"yoast_head":"\nHow to Optimise Website Crawlability | IMSR<\/title>\n<meta name=\"description\" content=\"Find out how to optimise your site & make it easier for Google crawlers to find the most important content. Rank higher on the SERPs.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Optimise Website Crawlability | IMSR\" \/>\n<meta property=\"og:description\" content=\"Find out how to optimise your site & make it easier for Google crawlers to find the most important content. Rank higher on the SERPs.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/\" \/>\n<meta property=\"og:site_name\" content=\"Improve My Search Ranking\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/ImproveMySearchRanking\" \/>\n<meta property=\"article:published_time\" content=\"2019-10-11T12:00:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-31T11:28:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.improvemysearchranking.com\/wp-content\/uploads\/2021\/11\/How-to-optimise-website-crawlability.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1217\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Samwise Diamond\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@ImproveMySearch\" \/>\n<meta name=\"twitter:site\" content=\"@ImproveMySearch\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Samwise Diamond\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/\",\"url\":\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/\",\"name\":\"How to Optimise Website Crawlability | IMSR\",\"isPartOf\":{\"@id\":\"https:\/\/www.improvemysearchranking.com\/#website\"},\"datePublished\":\"2019-10-11T12:00:55+00:00\",\"dateModified\":\"2024-10-31T11:28:13+00:00\",\"author\":{\"@id\":\"https:\/\/www.improvemysearchranking.com\/#\/schema\/person\/84666854cebd4e342f8033db55cc6c0a\"},\"description\":\"Find out how to optimise your site & make it easier for Google crawlers to find the most important content. Rank higher on the SERPs.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.improvemysearchranking.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Optimise Website Crawlability\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.improvemysearchranking.com\/#website\",\"url\":\"https:\/\/www.improvemysearchranking.com\/\",\"name\":\"Improve My Search Ranking\",\"description\":\"Improve My Search Ranking\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.improvemysearchranking.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.improvemysearchranking.com\/#\/schema\/person\/84666854cebd4e342f8033db55cc6c0a\",\"name\":\"Samwise Diamond\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.improvemysearchranking.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/601b5d268dd944bbba7c27b0af7fec06?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/601b5d268dd944bbba7c27b0af7fec06?s=96&d=mm&r=g\",\"caption\":\"Samwise Diamond\"},\"url\":\"https:\/\/www.improvemysearchranking.com\/author\/samwise\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Optimise Website Crawlability | IMSR","description":"Find out how to optimise your site & make it easier for Google crawlers to find the most important content. Rank higher on the SERPs.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/","og_locale":"en_GB","og_type":"article","og_title":"How to Optimise Website Crawlability | IMSR","og_description":"Find out how to optimise your site & make it easier for Google crawlers to find the most important content. Rank higher on the SERPs.","og_url":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/","og_site_name":"Improve My Search Ranking","article_publisher":"https:\/\/www.facebook.com\/ImproveMySearchRanking","article_published_time":"2019-10-11T12:00:55+00:00","article_modified_time":"2024-10-31T11:28:13+00:00","og_image":[{"width":1920,"height":1217,"url":"https:\/\/www.improvemysearchranking.com\/wp-content\/uploads\/2021\/11\/How-to-optimise-website-crawlability.jpg","type":"image\/jpeg"}],"author":"Samwise Diamond","twitter_card":"summary_large_image","twitter_creator":"@ImproveMySearch","twitter_site":"@ImproveMySearch","twitter_misc":{"Written by":"Samwise Diamond","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/","url":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/","name":"How to Optimise Website Crawlability | IMSR","isPartOf":{"@id":"https:\/\/www.improvemysearchranking.com\/#website"},"datePublished":"2019-10-11T12:00:55+00:00","dateModified":"2024-10-31T11:28:13+00:00","author":{"@id":"https:\/\/www.improvemysearchranking.com\/#\/schema\/person\/84666854cebd4e342f8033db55cc6c0a"},"description":"Find out how to optimise your site & make it easier for Google crawlers to find the most important content. Rank higher on the SERPs.","breadcrumb":{"@id":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.improvemysearchranking.com\/optimise-website-crawlability\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.improvemysearchranking.com\/"},{"@type":"ListItem","position":2,"name":"How to Optimise Website Crawlability"}]},{"@type":"WebSite","@id":"https:\/\/www.improvemysearchranking.com\/#website","url":"https:\/\/www.improvemysearchranking.com\/","name":"Improve My Search Ranking","description":"Improve My Search Ranking","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.improvemysearchranking.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/www.improvemysearchranking.com\/#\/schema\/person\/84666854cebd4e342f8033db55cc6c0a","name":"Samwise Diamond","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.improvemysearchranking.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/601b5d268dd944bbba7c27b0af7fec06?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/601b5d268dd944bbba7c27b0af7fec06?s=96&d=mm&r=g","caption":"Samwise Diamond"},"url":"https:\/\/www.improvemysearchranking.com\/author\/samwise\/"}]}},"_links":{"self":[{"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/posts\/10174"}],"collection":[{"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/comments?post=10174"}],"version-history":[{"count":1,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/posts\/10174\/revisions"}],"predecessor-version":[{"id":23293,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/posts\/10174\/revisions\/23293"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/media\/18043"}],"wp:attachment":[{"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/media?parent=10174"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/categories?post=10174"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.improvemysearchranking.com\/wp-json\/wp\/v2\/tags?post=10174"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}