How to Generate Sitemap in Magento: Understanding What Google Wants

A sitemap is the simplest way to inform Search Engines about new pages of the site which are available for scanning. The simplest sitemap form is a XML file in which all the addresses (URL) of a certain site are shown with a short description of this site or without it. Metadata which is used for the design of this page helps the search engines to scan the pages faster and more correctly.

Setting a sitemap.xml properly helps search engine bots:

• understand which pages are new on the web resource and need to be indexed
• prioritize pages depending on the specified parameters
• find out when the last changes were made to the pages that have already been indexed;remember the frequency of update pages
• index all the necessary pages, without exception, because if there is no site map and the research is done blindly, there is a high probability that it will simply miss some of them.

In other words, a sitemap is something that can be a kind of a guide to your website, which greatly “simplifies life” for search engine robots and thus increases their level of trust. And this point also adds to the web resource some very useful additional rating points.

How to Create Sitemap in Magento

As a rule, the sitemap helps to find all the internal site links and sometimes it helps to find the external ones and it is very which is very useful. You can bind several pages of a site to each other with Metadata, compute the necessary URL for indexing and many others.

The main thing you need to know about the sitemap is that its presence does not guarantee that the site will be added to the top of search results by the certain requests or the page will be indexed necessarily. But the existence of this file will become a certain tip for the search engines which will help to improve their work in your Magento website.

This card has become widespread in Google Search Engines, Yahoo! (since 2011 this search engine has changed the requirements to the standard card view), and Microsoft. The majority of webmasters think that Google sitemap also helps to index the site with other search engines.
You can get to know the detailed description of the requirements to this file on the sitemaps.org.

How to Create Sitemap in Magento

If you are using the sitemap to provide search engine spiders with a simple system of getting around your project then creating them within an XML platform would be best.
However, if you want to provide this process as yet another way for people to navigate your site then having an HTML sitemap is highly recommended.
Given that both options have their own advantages to your eCommerce project we recommend considering both rather than either or. Depending on what you decide as your need it will directly influence which of the two you opt into.

HOT TIP: When considering sitemaps – if you release videos on a regular basis consider implementing a sitemap that includes listing videos. If you do decide to use this type of a sitemap it should not be done as a replacement for either of the other two suggestions but rather something to be augmented with other navigational properties.

The main advantage of e-commerce Magento platform over other CMS is that it has the ability to create the sitemap auto mode, but not with the integration of the special file in the root system. In order to do it, you should go to your admin panel and find the Top Menu section – Catalog. You will find a special item in this section called Google Sitemap. You should choose the Add Sitemap item there. Then enter the name of your Sitemap and the necessary format which is XML in our case.

The Sitemap saving path is specified automatically, but if it is necessary you can indicate it manually. For example, you need to save the Sitemap in some subsection of the main menu. Then you need to write the following path:

Your subsection / Sitemap (Be sure to place diagonal slash).

Please check whether the chosen section is indexed. After this, you should choose the Store View item and click on the Save Sitemap button.

Using such a simple algorithm you will get an automatic sitemap which will contain the links to all the categories, commodity positions and other pages of your online store. All these items can be chosen in the appropriate position. Also, Magento ® edits your file with the sitemap itself while adding or removing any pages to/from the site. In order to configure this function you need to go to the admin panel: System – Configuration – Google Sitemap.

You can get to know the detailed description of the requirements to this file on the site http://www.sitemaps.org

Should I Use HTML or XML Sitemaps With My Magento ® Storefront

If you are using the sitemap to provide search engine spiders with a simple system of getting around your project then creating them within an XML platform would be best.

However, if you want to provide this process as yet another way for people to navigate your site then having an HTML sitemap is highly recommended.

Given that both options have their own advantages to your eCommerce project we recommend considering both rather than either or.  Depending on what you decide as your need it will directly influence which of the two you opt into.

HOT TIP:  When considering sitemaps – if you release videos on a regular basis consider implementing a sitemap that includes listing videos.  If you do decide to use this type of a sitemap it should not be done as a replacement for either of the other two suggestions but rather something to be augmented with the other navigational properties.

How to create a Magento Sitemap

The main advantage of ee-commerceMagento ® platform over other CMS is that it has the ability to create the sitemap automatically, but not with the integration of the special file in the root system. In order to do it, you should go to your admin panel and find Top Menu section – Catalog. You will find a special item in this section called Google Sitemap. You should choose the Add Sitemap item there. Then enter the name of your Sitemap and the necessary format which is XML in our case.

The Sitemap saving path is specified automatically, but if it is necessary you can indicate it manually. For example, you need to save the Sitemap in some subsection of the main menu. Then you need to write the following path:

Your subsection / Sitemap (Be sure to place diagonal slash).

Please check whether the chosen section is indexed. After this, you should choose the Store View item and click on the Save Sitemap button.

Using such a simple algorithm you will get an automatic sitemap which will contain the links to all the categories, commodity positions and other pages of your online store. All these items can be chosen in the appropriate position. Also, Magento edits your file with the sitemap itself while adding or removing any pages to/from the site. In order to configure this function you need to go to the admin panel: System – Configuration – Google Sitemap.

Magento Sitemap Generation Settings

After creating the file, you need to add it to the GSC (Google Search Console) and robots.txt.
For different types of content (video, images, pages), you need to use different sitemaps (on the Google side, different crawlers are used and a specific syntax is used for each content).
In order for robots not to “stumble” while viewing a map of your web resource, you need to know how to set up sitemap.xml correctly and what are the necessary conditions:

• No file encoding other than UTF-8 is allowed.
• The file size should be up to 10 MB.
• 50,000 URLs – is the limit of addresses number of pages in the sitemap.xml body.
• When one of the previous two indicators is exceeded, the sitemap.xml setup involves breaking it into several files.
• The file should be stored in the root folder of the web resource.

How to Create a Magento Sitemap: Tell Search Engines about Sitemap

To inform Search Engines about your Sitemap for Magento you can bind it to the file called robots.txt.
You need to go to this file and edit it. You should write the following line in this file: Sitemap.

To register a new Sitemap you should go to the websites control panel for webmasters in Google Search Engine. If you do not want to send the Magento Sitemap yourself, you can use special services or contact GoMage specialists in order to get any assistance with your questions.
Also, you might be interested in the SEO extension for Magento which improves search engine optimization. It’s easy to use and there is no need for any technical knowledge.

Magento Sitemap Generator: Customizing Sitemap.xml

If you want to configure sitemap.xml you may use one of the following steps:

• Manually. It is a rather difficult job, which you can do yourself if your web resource consists of a small number of pages. It implies the presence of skills to work with the main tags.
• Customize sitemap.xml using specialized online services. There are both paid and free options. All of them work according to a similar principle – you specify the main address of your site, the initial settings and get the generated result. Most often, they have a limit of up to 5,000 page addresses per file.
• Using generators. Setting up sitemap.xml with their help has a number of advantages (the maximum possible number of addresses included, a large number of settings) and one slightly negative point – almost all “serious” options are supplied subject to license purchase (it costs from $ 20).

Magento Sitemap Generator: Customizing Sitemap.xml

• Use the same syntax while specifying a URL. Google will scan them exactly on the list. For example, if the site is located at http://www.example.com/, do not use the URL /https://example.com/ (without www) or ./mywebsite.html (relative URL).
• Do not include session IDs in the URL being added – this can lead to excessive page crawling.
• Tell Google about the version of the page in other languages ​​using the hreflang attribute.
• Sitemaps must use UTF-8 encoding, and the URL must be escaped.
• Split large Sitemaps into smaller files to reduce the load on the server when sending information to Google. Create a Sitemap index file, listing all the Sitemaps in it, and submit it to Google instead of individual files.
• Use the recommended URL canonicalization methods to tell Google about the version of the domain for which the Sitemap was generated (if the site is available with or without www). In this case, only one Sitemap is required.
• Use Sitemap extensions for additional types of information, such as videos, images, and news.
• If the mobile and regular versions of the page have different URLs, we recommend pointing out only one of them. If you still need to point to both URLs, add annotations for them so that different versions are recognized.
• If there are versions of the page in other languages ​​or for other regions, you can specify them using a Sitemap or hreflang attribute.
• Use only numbers and Latin letters. The sitemap must be in UTF-8 encoding (you can usually choose it when saving). A sitemap can contain only ASCII characters. It should not contain extended ASCII characters, certain control codes, and special characters such as * and {}. If they are contained in a URL, an error message will appear when you try to add it.

In addition, all URLs (including addresses in your Sitemap file) must be properly shielded and coded so that they can be processed by the web server. This is usually done automatically if you use a script, tool, or log file to create the URL, that is, do not enter addresses manually. If you receive a notification when Google sends your Sitemap that Google could not find your URLs, make sure they comply with the RFC-3986 URI standard, the RFC-3987 IRI standard, and the XML standard.

Magento Sitemap Generation Settings: Frequent Errors in the Sitemap & Techniques to Check Them

Errors will not have a critical impact on the attitude of the search engines to your site. However, their removal will allow the released resources of the search robot to redirect important pages.
The most common mistakes:

• broken links;
• redirects;
• incorrect dates for publishing or updating content.

To verify the correctness of the site map, numerous online services have been developed that will do it for you automatically and point out all the errors that have been made:

• the validity of the file
• an indication of the URL from another domain
• the excess file size
• online diagnostics of the indexing.

Such programs are free and work with all major search engines (Google, Yandex, Yahoo).
To check the correctness, you need to open the service page, load the XML-file or enter the corresponding URL in the field and click on the check button. If the program detects some errors, it will prompt which addresses give an incorrect answer.

How to Generate Sitemap in Magento: Creating and Sending Sitemaps

• Decide which pages of your site your Google search robots should crawl and determine the canonical URLs for each of them.
• Determine which Sitemap file format to use. You can create Sitemaps manually or automatically using third-party tools.
• Make your sitemap available to Google. To do this, add a link to it in the robots.txt file or send it directly to Google using the Search Console.

Sitemap Formats. Google supports several Sitemap formats, described below. All formats should use a standard protocol. Google currently does not support the attribute in Sitemaps. If the file size or the number of addresses listed in it exceeds the limits, break it into several parts. You can create a Sitemap index file by listing all the Sitemaps in it, and send them to Google.

In the conclusion, it is worth to say that the correct setting of a sitemap.xml can have a very positive effect on the speed of indexation of the pages of your web resource and the general opinion of search engines about it. Therefore, we strongly recommend you to create a map for each of your websites, work with the customization and monitor its timely updates. If you are still not sure in your own knowledge you can always hire some professionals who will organize all the necessary points for you and your resources.