.Google's John Mueller responded to a question about why Google.com indexes web pages that are forbidden from creeping through robots.txt and also why the it is actually risk-free to ignore the associated Search Console files about those crawls.Robot Traffic To Concern Parameter URLs.The person talking to the question documented that bots were creating hyperlinks to non-existent concern parameter URLs (? q= xyz) to pages with noindex meta tags that are actually additionally blocked out in robots.txt. What triggered the inquiry is that Google is crawling the web links to those web pages, getting shut out by robots.txt (without envisioning a noindex robotics meta tag) after that acquiring reported in Google.com Look Console as "Indexed, though blocked by robots.txt.".The individual talked to the adhering to inquiry:." But listed here is actually the big question: why would certainly Google.com mark web pages when they can not also see the information? What's the perk because?".Google.com's John Mueller validated that if they can not crawl the web page they can't find the noindex meta tag. He also creates an interesting mention of the website: search operator, suggesting to ignore the results given that the "common" individuals will not view those results.He wrote:." Yes, you're proper: if our company can't crawl the page, our company can't observe the noindex. That stated, if we can't creep the pages, at that point there is actually certainly not a great deal for our team to mark. So while you may see several of those pages with a targeted web site:- concern, the average user will not view them, so I wouldn't bother it. Noindex is additionally fine (without robots.txt disallow), it simply suggests the Links will certainly wind up being actually crawled (and find yourself in the Look Console file for crawled/not listed-- neither of these conditions lead to problems to the rest of the web site). The integral part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the limits in using the Website: search accelerated search operator for diagnostic explanations. One of those main reasons is due to the fact that it is actually certainly not connected to the routine hunt index, it is actually a distinct thing entirely.Google's John Mueller discussed the web site search driver in 2021:." The quick response is actually that a website: concern is actually not suggested to be total, nor used for diagnostics functions.A site query is a details kind of search that confines the end results to a certain site. It's essentially only words internet site, a digestive tract, and after that the web site's domain.This query limits the results to a certain website. It's not indicated to be a thorough assortment of all the webpages coming from that web site.".2. Noindex tag without using a robots.txt is actually alright for these kinds of circumstances where a bot is connecting to non-existent web pages that are receiving found through Googlebot.3. Links along with the noindex tag will generate a "crawled/not recorded" entry in Explore Console and also those will not possess a negative impact on the rest of the web site.Check out the question and also answer on LinkedIn:.Why would certainly Google index pages when they can not also find the content?Featured Photo through Shutterstock/Krakenimages. com.