If you often encounter indexing problems and don't see some of the pages in the search, it's time to check the structure of your resource. A sitemap will help you with this, and search robots use it to check website pages. In this article, we'll look at the sitemap.xml file type and learn how to create and customize it.

What is a sitemap

A sitemap, or sitemap, is a list of pages on a site in a hierarchical order. Used by search engines - in XML format, or to facilitate navigation on the resource - an HTML file.

XML – is checked by robots that index the site's pages and its structure.

HTML – a page with links to all sections of the resource. Namely, the hierarchical sequence of sections, categories, and pages. It was created for users to make it easier to find the desired site area.

This format is rarely used, but a similar structure can still be found in the site's footer. Here are the main sections of the resource, but without additional nesting.

Learn more: Magento 2 development: PRO level

Why do you need a sitemap?

A sitemap is a navigator that helps search engines correctly crawl a site and index its pages.

Is it possible to do without a sitemap?

The sitemap is the recommended tool. The Yandex search robot will mark this as a possible but not a critical problem if you do not have a sitemap.

Although algorithms are getting smarter and can independently index the site, maps are an indispensable assistant in certain situations. For example, this is true for multi-page and complex sites with tens of thousands of attachments.

Sitemap needed:

- New sites with few external links.

- For those who often make changes.

- Resources that have priority and less significant pages.

- Sites with a complex structure.

- If there are pages on the site that are not connected to others, they cannot be accessed through an internal or external link.

- Sitemap also helps to track down several issues quickly:

URL crawling errors.

- Pages closed from indexing.

- Error 404.

Learn more: How to get an SSL certificate for free and why you need it

What are sitemaps

Now let's look at the types of an XML sitemap. The HTML map has only one view, which we discussed at the beginning of the article.

Information about other sitemaps in a single file

All site links are listed in several documents: sitemap.xml contains other maps. Such maps are used to separate an extensive list of URL pages, and it applies, for example, to corporate websites and online stores.

What should be in a sitemap

And now, let's understand the parameters, requirements, and sitemap tags.

sitemap.xml requirements

1. File encoding - UTF-8. A common Unicode character encoding standard that allows more compact storage and transmission of information.

2. The maximum number of links in one file is 50,000.

3. File size - up to 50 megabytes.

4. Links correspond to one host - domain and primary mirror: example.com or store.example.com.

5. The request to the server returns the response, "The HTTP 200 OK."

Where to embed sitemap.xml

The sitemap must be placed in the root folder of the site.

Add a sitemap directive to the robots.txt file. Example of an entry in a document: "Sitemap: http://site.com/sitemap.xml."

Where to embed sitemap.html

1. The map should not have more than 200 links. If you need to add more - use a multi-stage card.

2. The map is usually linked in the site's footer or on page 404. By clicking on the link, the user goes to a separate page where the list of all resource pages is located.

3. The sitemap.html directive is also added to the robots.txt file.

4. As anchors, use relevant words related to the page's content.

5. The map must be updated as changes are made to the site, so it does not lose its relevance.

6. HTML map must match the structure of the site - sections, categories, pages, and attachments.

Create a sitemap

And in this section, we will look at options for creating a sitemap.

Online generator

In online services, you can quickly create a map. Paid generators allow you to work with large sites, free ones - with landing pages. You can use the services offered in this article or find a more convenient one - there is a wide range of options on the network.

My Sitemap Generator

My Sitemap Generator is one of the popular generators. It can process up to 500 URLs for free. If you need more, you need to purchase a subscription - from $ 2.5.

The generator allows you to customize the processing of the site.

XML Sitemap Generator

XML Sitemaps Generator is an easy-to-manage service: enter the site address and get a map. It allows you to process 500 pages free from 1,000 URLs; a subscription costs $4.99 per month. The paid version also allows you to work with the site and check SEO - resource structure, and external and internal links.

Via software

You can find hundreds of programs on the Internet that help you crawl your site and check SEO, such as Sitemap Writer Pro or SiteMap Generator. We will talk about Screaming Frog SEO Spider Tool, a free search engine optimization service. With the help of this software, you can comprehensively audit the site.

1. Download and install the app first.

2. In the search bar, enter the URL of the site and click "Start." Let the software check the resource.

3. After that, find the "Sitemaps" section in the top menu and select the appropriate "XML Sitemap" or "Images Sitemap."

How to check sitemaps for errors

The sitemap is checked to rule out problems with resource indexing. For example, if a random page began to give an error or was previously deleted by you. After the changes are made, you need to update the sitemap.

If you have a small site and rarely update the page information, doing a scheduled check every six months is enough. And if you constantly change or add data to the resource, you need to check the map more often and submit new pages for indexing.

Screaming Frog SEO Spider Tool

Find the "Mode" section in the top menu and switch the scan type to "List."

An Upload button will appear in the middle. Click and select "Download XML Sitemap."

Recommendations - how to work with sitemap.xml

1. Hide your sitemap to protect unique pages. There are more and more services for parsing. If a competitor completely copies the structure of your site and clones it, you may lose the uniqueness of your resource, and the potential of the original pages in search results will decrease.

2. Create images, videos, and news maps. Such cards will solve the problem with indexing - robots will understand the content on your site faster and better. Online stores often face similar difficulties - in a situation where the bot cannot recognize all the images of the goods. Maps will also help you upload graphical content to the relevant sections on photo and video searches.

3. Collect small cards into one big one if you have a site with thousands of URLs.

4. Don't forget to add the map to the robots.txt file and the webmaster panel.

5. Create a dynamic sitemap if you have an aggregator site with many pages on which information is constantly updated. The robot checks the sitemap - indexes the pages presented on the map. You delete already checked pages and load new ones into the map that need to be updated. Thus, the robot actively contains all site pages and does not miss anything.

6. Creating a map is optional if you have a landing page.