An XML sitemap is a file that lists the URLs for a website to help search engines crawl and index its pages more efficiently. A complete sitemap includes all the publicly available pages on a site and provides valuable information about each URL to search engine crawlers.
This comprehensive guide will explain what an XML sitemap is, the benefits of using one, how to create and optimize a sitemap, and tips for maintaining it.
What is an XML Sitemap?
An XML sitemap is a file that contains a list of all the pages on a website along with additional metadata about each URL. The sitemap uses XML markup language and conforms to the Sitemaps protocol, which was created by Google, Yahoo, and Microsoft.
The purpose of an XML sitemap is to provide search engines with information about the organization and content on a website. It helps the crawlers index new or updated pages faster and more accurately. The sitemap file is submitted to search engines like Google, Bing, and Yahoo to notify them of all the URLs they should crawl.
Benefits of Using an XML Sitemap
Here are some of the key benefits of having an XML sitemap:
- Improves indexation of new pages - When new pages are added to a site, an updated sitemap helps search engines discover these URLs faster for indexing.
- Recovers lost pages - If pages drop out of the index due to a site redesign or redirect, a sitemap helps search engines recrawl those URLs.
- Provides metadata for pages - The sitemap allows you to include details like when a page was last updated, how often it changes, and its relative importance. This helps search engines determine crawl frequency and priority.
- Faster crawling and indexing - By listing all pages in one place, the sitemap reduces the amount of time search engines need to crawl a site. Pages are discovered faster.
- Supports SEO audits - Sitemaps provide a handy list of all URLs when auditing sites and identifying issues to fix.
- Helps with site migrations - After moving a site to a new domain, a sitemap helps ensure all pages from the old domain get crawled and indexed under the new domain.
How to Create an XML Sitemap
Here are the key steps involved in creating and configuring a basic XML sitemap:
1. Identify all publicly available pages to include. This encompasses all important content like blog posts, product pages, categories, tags, and pages in site navigation. Exclude pages blocked by robots.txt file.
2. Determine the site structure - Whether it has a simple structure starting from the homepage or a complex structure with layers of subdirectories.
3. Choose a sitemap builder tool or plugin for your CMS platform. Popular options for WordPress include Google XML Sitemaps, XML Sitemap & Google News feeds, and Yoast SEO.
4. Configure the sitemap settings - Indicate URL paths to include/exclude, update frequency, priority levels, last updated timestamps, etc. Most sitemap builders automatically pull this data.
5. Generate the XML sitemap file. It will have a .xml extension by default.
6. Submit the sitemap to search engines through Google Search Console, Bing Webmaster Tools, and Yahoo Site Explorer.
7. Set a schedule for search engines to crawl the sitemap file periodically as content is updated on the site.
8. Update the sitemap whenever new pages are published and resubmit to the search engines. Automate this process if possible.
XML Sitemap Structure and Tags
The XML sitemap must adhere to a defined structure and use standard XML tags. Here are the key components:
- XML declaration and root <urlset> tags - Indicate this is an XML sitemap document.
- <url> tag - Denotes the start of data for each URL in the sitemap.
- <loc> tag - The actual URL listed. This is the only required tag per URL.
- <lastmod> tag - Date when the page was last updated.
- <changefreq> tag - How frequently the page changes - always, hourly, daily, weekly, monthly, yearly, never.
- <priority> tag - The relative priority of the page compared to others on the site, from 0.0 to 1.0.
- <sitemapindex> tag - For listing sitemap files if creating a sitemap index file.
Here is an example of how the code looks for 2 URLs:
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/page1.html</loc>
<lastmod>2023-01-25</lastmod>
<changefreq>monthly</changefreq>
</url>
<url>
<loc>http://www.example.com/page2.html</loc>
<lastmod>2023-01-23</lastmod>
<changefreq>weekly</changefreq>
</url>
</urlset>
Best Practices for Optimizing XML Sitemaps
Here are some key best practices to follow when creating and configuring your sitemaps:
- Keep the file size under 50MB and with fewer than 50,000 URLs. Split into multiple sitemaps if exceeding this.
- Incorporate all page types - HTML, PDF, images, videos, etc. Use video and image sitemap extensions as needed.
- Update sitemap frequently as new pages are added - daily or weekly at minimum.
- Set reasonable change frequencies based on content types. For example, assign weekly for blog posts and yearly for static pages.
- Give higher priority to important pages like the homepage, product categories, contact pages, etc.
- Use the "lastmod" date strategically to show newly added or updated content.
- Include canonical version of pages only, no duplicates.
- Add sitemap URL to your robots.txt file to help search engines discover it.
- Compress the sitemap using GZIP to enhance crawl efficiency.
- Prevent access to the sitemap file on your site, only submit it to search engines.
- Check for errors - validate the format using online tools like the W3C validator.
- Monitor indexation in search engine webmaster tools and tweak accordingly.
- Create separate sitemaps for images, videos, and news content if you have an extensive media library.
Tips for Maintaining XML Sitemaps
Here are some tips for keeping your sitemaps updated and managed properly as part of an ongoing SEO strategy:
- Use a sitemap generator tool to automate creation and pinging search engines when updated.
- Schedule automated cron jobs to run daily or weekly to detect changes and generate fresh sitemaps.
- When migrating to a new domain, create a new sitemap and submit to search engines after establishing redirects.
- Check regularly for broken links using crawling tools and remove or fix these pages.
- Monitor search engine webmaster tools for crawl errors and troubleshoot as needed.
- Audit sitemaps periodically to remove outdated, unimportant, or duplicate pages.
- Create a reminder to regenerate and resubmit your sitemaps on a quarterly basis.
- If you have multiple sitemaps, implement master sitemap index file for easier management.
- When removing a page, update the sitemap first before deleting the URL to avoid crawl errors.
- Review sitemaps after making major site architecture changes or integrating a new CMS.
Creating and maintaining XML sitemaps take some effort but are well worth it for the search visibility and traffic benefits they provide. Follow this comprehensive guide to get started with sitemaps and use them effectively to take your SEO efforts to the next level.