.Google.com's John Mueller addressed an inquiry regarding why Google.com indexes webpages that are refused from creeping by robots.txt and also why the it's risk-free to overlook the related Look Console documents regarding those creeps.Bot Traffic To Query Parameter URLs.The person asking the inquiry chronicled that crawlers were actually generating links to non-existent concern parameter Links (? q= xyz) to web pages along with noindex meta tags that are also obstructed in robots.txt. What cued the question is actually that Google.com is actually crawling the links to those pages, getting obstructed through robots.txt (without noticing a noindex robots meta tag) then obtaining turned up in Google Search Console as "Indexed, though blocked through robots.txt.".The person inquired the adhering to question:." But listed below is actually the major concern: why will Google mark web pages when they can not also observe the information? What's the advantage during that?".Google.com's John Mueller validated that if they can't crawl the webpage they can not view the noindex meta tag. He likewise creates an appealing acknowledgment of the website: search driver, suggesting to dismiss the results since the "typical" customers won't find those results.He created:." Yes, you are actually appropriate: if we can't crawl the page, our team can't find the noindex. That said, if our company can not creep the pages, then there's not a lot for us to mark. So while you may view a few of those pages along with a targeted site:- inquiry, the ordinary user won't observe them, so I wouldn't fuss over it. Noindex is likewise fine (without robots.txt disallow), it only suggests the URLs will certainly end up being actually crept (and also find yourself in the Explore Console document for crawled/not listed-- neither of these statuses result in issues to the rest of the website). The vital part is actually that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the restrictions in using the Website: hunt advanced search driver for analysis explanations. Some of those causes is since it's certainly not linked to the normal hunt index, it's a separate point completely.Google's John Mueller commented on the internet site hunt driver in 2021:." The short solution is actually that a website: concern is actually certainly not indicated to be total, neither used for diagnostics reasons.An internet site query is actually a certain sort of hunt that confines the results to a certain web site. It is actually generally simply the word internet site, a colon, and afterwards the site's domain.This inquiry confines the outcomes to a particular web site. It's certainly not meant to be a thorough selection of all the webpages from that internet site.".2. Noindex tag without using a robots.txt is alright for these sort of circumstances where a crawler is actually linking to non-existent pages that are actually receiving found out by Googlebot.3. URLs with the noindex tag are going to generate a "crawled/not indexed" item in Search Console which those will not possess an adverse result on the rest of the web site.Review the question and address on LinkedIn:.Why would certainly Google mark pages when they can't even see the content?Featured Image through Shutterstock/Krakenimages. com.