Google Has a Separate User Agent For Crawling Sitemaps & For GSC Verification
Google has a separate user agent that fetches the sitemap file, as well as one to crawl for GSC verification. John recommends making sure you are not blocking these. The post Google Has a Separate User...
View Article“Discovered Not Indexed” Pages May Show in GSC When Only Linked in Sitemap
Pages may show as “Discovered Not Indexed” in GSC if they have been submitted in a sitemap but aren’t linked to within the site itself. The post “Discovered Not Indexed” Pages May Show in GSC When Only...
View ArticleUse the Last Modified Date to Provide a Hierarchy of Changes Made to A Site
John recommends using the last modified date in sitemaps in a reasonable way to provide a clear hierarchy of the changes that have been made on a site. This helps Google to understand which pages are...
View ArticleInclude Most Recently Changed Content in Separate Sitemap
Rather than submitting all of your sitemaps regularly to get Googlebot to find and crawl newly updated pages, John recommends adding recently changed pages into a separate sitemap which can be...
View ArticleSpecify Timezone Formats Consistently Across Site & Sitemaps
Google is able to understand different timezone formats, for example, UTC vs GMT. However, it’s important to use one timezone format consistently across a site and its sitemaps to avoid confusing...
View ArticleUse Sitemaps Ping, Last Modified and Separate Sitemaps to Index Updated Content
To help Google index updated content more quickly, ping Googlebot when a Sitemap has been updated, use Last Modified dates in Sitemaps, and use a separate Sitemap for updated content so it can be...
View ArticleImage sitemaps can be useful for sites that use lazy loading
When “lazy loading” images on a page in a way that doesn’t include defined image elements, it’s recommended to have back-up in the form of structured data or an image sitemap. That way, Google will...
View ArticleRobots.txt file size doesn’t impact SEO, but smaller files are recommended
John confirmed that the size of a website’s robots.txt file has no direct impact on SEO. He does, however, point out that larger files can be more difficult to maintain, which may in turn make it...
View ArticleIt’s possible to host sitemap files on a separate domain
One user asked whether they could host their sitemap files externally (perhaps on a separate server or a staging site). John explains that yes, that’s possible as long as the sitemaps are handled...
View ArticleIt’s okay if the same URL appears on multiple sitemap files
It’s fine to have the same URL included in multiple sitemap files. The only caveat is ensuring that there is no conflicting information being provided across the different sitemaps. For example, having...
View Article