Recently, we started experiencing indexing issues on multilingual websites — both those developed by us and those where we were asked to intervene.
While researching the problem, we noticed that many users had signalled similar challenges. Therefore, I decided to share the definitive solution here, hoping it may be helpful to others.
In this article, I’ll also discuss best practices for indexing multilingual websites on Google and go through all the steps that led us to resolve the issue.
If you prefer to jump straight to the solution, you can scroll down to the section titled “The Definitive Solution to Indexing Problems on Multilingual Websites”.
Google provides useful guidance on how to manage multilingual websites on the following page::
These guidelines should be considered on a case-by-case basis, but for most projects starting from scratch, the simplest solution is to use a generic top-level domain (such as .com) and organize the languages using language-specific subfolders (/de, /es, /fr, etc.).
example: https://mysite.com/es/contenido-traducido-en-espanol
Compared to other solutions, this approach offers the following advantages:
Easy to set up
Leaner structure
More cost-effective for the client
Generic top-level domains (gTLDs), particularly .com domains, are generally better suited than country-code domains (.it, .de, .es, etc.) in cases where a company does not want to provide just a translation of its content for foreigners located in its own country, but aims to rank the website internationally (i.e., a multi-regional website).
After extensive testing, for several years we have chosen Polylang as the plugin for building multilingual websites, as it allows for intuitive and efficient language management without any extra costs.
Here are the features that made us choose Polylang over other multilingual plugins:
Easy and quick to set up
Ability to choose language plus country codes
Intuitive and efficient translation management, even for clients who will manage the site independently
Ability to translate any type of element (templates, widgets, taxonomies), leaving no elements untranslated
SEO-friendly
Availability of a dedicated plugin for integration with Elementor page builder
Since 2022, to speed up the indexing of the websites we publish, we have implemented an integration between Google Search Console and Google Cloud using the Instant Indexing API, via a plugin developed by Rank Math.
This approach has resolved indexing delays from the search engine, which in some cases previously took weeks or even months to detect and index all pages of newly published websites, especially those with more than the typical 5–6 pages.
This year, we decided to permanently replace the well-known Yoast SEO plugin with Rank Math. While Yoast remains an excellent SEO plugin, my impression is that it has recently fallen behind in implementing new features, particularly regarding SEO process automation and optimization for AI-driven search.
The Rank Math team has been doing an outstanding job in recent years, starting with being the first to provide Instant Indexing, through a separate plugin from their main SEO plugin.
Rank Math features a lighter codebase and simplifies SEO workflows by including, even in the basic version, functionalities that previously required third-party plugins, such as:
AI-generated metadata
Dynamic generation of Alt attributes
Redirection manager
404 error logging
Content indexing blocks
Ability to edit the .htaccess file
Local SEO settings
Moreover, Rank Math offers more advanced AI optimization, allowing for richer company data integration across different sections and generating a much more comprehensive llms.txt file.
Despite all the measures mentioned above, we recently noticed that some multilingual websites were slow to be indexed and to rank on Google, even when searching for the brand name.
The issue did not occur on other search engines such as Bing or DuckDuckGo.
Moreover, when searching for the brand or domain on Google, the sitelinks were slow to appear as a feature on the SERP.
We quickly realized that the websites affected by the issue shared the following characteristics: a .com domain with Italian as the primary language and /en as a subdirectory for the English translation.
Since the issue only occurs on Google and not on other search engines, we concluded that the problem stems from the search engine not recognizing which language is the primary one.
In 2022, Google discontinued the International Targeting tool within Search Console.
The tool allowed webmasters to specify which country or language their site was intended for.
As a result, proper indexing of multiregional websites now relies on hreflang tags — which for WordPress sites are configured by multilingual plugins such as Polylang — as well as on sitemaps and URL structures.
Since the International Targeting tool was no longer available to set and correct targeting issues, we resolved the first cases of indexing problems by manually configuring canonical tags, even though this step should not be necessary, as SEO plugins typically handle it automatically.
More recently, however, this solution did not work on new sites affected by the issue and, in fact, caused missing URLs in the sitemap of a multilingual site translated into three languages.
Initially, we believed the issue was due to compatibility between the SEO plugin and the multilingual plugin. Indeed, Rank Math states on its website that Polylang does not have native compatibility with Rank Math and offers a solution via PHP code, which can be found on this page: https://rankmath.com/kb/polylang-compatibility/
However, the issue occurred in the same way on sites using Yoast as the SEO plugin.
Before resorting to coding solutions, we decided to try a simpler approach first.
To definitively resolve the indexing issue, it was sufficient to uncheck the option “Hide language information for the default language in URL” within the “URL Modifications” section of the Polylang settings.
This way, the language code (/it) was also included in the Italian URLs and on the homepage, consistent with the setting “The language is set from the directory name in pretty permalinks” in the same settings section.
This results in a less “clean” appearance for the URLs of the main language, particularly the homepage, which now includes the language code even for Italian. However, aesthetic preferences naturally take a back seat to the necessity of correct indexing.
In fact, many major companies —whether using custom-built sites or CMS-based platforms— have adopted this solution (e.g., Ferrero, Essilor Luxottica), sometimes even including the country code alongside the language code in the URLs, e.g., hreflang=”en-us” (Ferrari, Barilla, Pirelli, etc.).
To accelerate indexing after adding the language code to the URLs, we resubmitted all the Italian pages using Instant Indexing, deleted and re-uploaded the sitemap in Search Console and… BANG! — the very next day all URLs were live, and the site ranked first when searching for the brand name. Within just a few days, sitelinks appeared as well. Problem solved.
For multilingual websites, it is also recommended to disable the option to automatically redirect users based on their browser language. While this feature may seem useful, it might also redirect Google crawlers —which typically access websites from the United States— preventing them from analyzing pages in languages other than English and potentially causing further indexing issues.
In the case of multilingual websites with generic top-level domains (such as .com) and languages managed through subdirectories, it is necessary to include the language code in all URLs, including the default language, especially if the default language is not English.
Special thanks to Mickey D’Iorio for playing a key role in identifying the definitive solution to the indexing issue.
