The 500 errors (Internal Server Error) is a general HTTP status code that means that the website server has had some kind of problem loading the web page, although it cannot specify what this problem is, as it is unknown.
Error 500 errors are primarily caused by a communication error between the browser and the server, and can occur for a number of common causes.
This problem can have many different causes, ranging from a problem with the Internet connection from the user’s Internet connection to lack of permissions in the browser, or even a power outage on the servers where the website is hosted. Similarly, this error has many possible solutions; from refreshing the browser to clearing the browsing history, cookies and cache to problems with the .htaccess file.
Despite being relatively straightforward problems, these can affect indexing of web pages and websites, since, if when Google’s “spiders” go to scan a website they find an error 500, it is possible that they will label the site as “unreliable” and will not analyze it with the correct periodicity, so its content will not be indexed correctly.
To find out if a website has problems with 500 errors, it is possible to access Google Search Console’s indexing reports, where all errors such as 200, 300, 400 and 500 will be displayed. For Google, any of these errors is an indicator that “something is wrong” with the site, so it is ideal to fix any of them as quickly as possible.
Luckily, the 500 errors are categorized as “soft errors” so Google doesn’t penalize to the pages or websites that suffer them. On the contrary, when Googlebot wants to analyze a page and encounters a 500 error, instead of tagging the error, what it does is to wait and come back to it later.
If after several attempts, Google’s spiders encounter the same error over and over again, then what they do is increase the time between each scan. If the error persists, then Google removes the web page from indexing.
Google believes that if a large portion of the pages that make up a website have 500 errors over an extended period of time, the problem may be a misconfiguration of the Robots.txt file in which the Googlebot is being blocked from accessing.
For those users who want to learn more about how Google deals with 500 errors on websites, below is John Mueller’s full user talk with users in which he provides details on this and other issues:
