Isn’t it crazy when pages that should be blocked by robots.txt end up in search results anyway? During this week’s Wednesday Workshop, we break down just how and why this happens.

Welcome back to another Wednesday Workshop from Happy Hao Car.

Last month, during our video on Technical SEO , we discussed that Google will no longer support robots.txt files that contain noindex directives starting September 1st. Rather than using robots.txt to block pages from being crawled, implementing noindex in robots meta tags is a better option.

But when you’re using robots.txt to hide a page from search engines, have you ever noticed that sometimes it gets indexed and ranked anyway? Today, we are going to investigate why this happens.

Google has said that it cannot index content that is blocked by robots.txt, and will typically index more accessible pages first; but it does its best to determine how relevant the content is to a search query through a couple of methods.

First, Google will compare a page’s URL with other URLs to find similarities and consistencies that may be descriptive of page content. However, this can still be difficult when a page is set to noindex.

Then, Google will evaluate the sources and types of links coming to the robotted page. In doing this, they are trying to establish relevance as it pertains to the search query.

If Google determines through backlinks to the blocked page that the content could be relevant, it may then display the page in search results; and if people are linking to the robotted content, Google feels it may be worthwhile for other users.

So what does this mean for you?

You need to limit the type of content that you block with your robots.txt file, as Google may choose to guess what it is and rank it anyway. Guessing probably won’t work out to your benefit! If it is something you truly need to have omitted from search, implementing noindex in robots meta tags is one of the best ways to prevent indexation.

That’s all the time we have left for today’s workshop. We covered a very challenging topic today; so feel free to leave your questions and comments down below, and we’ll get back to you shortly. Thanks for watching. We’ll see you next week with another Wednesday Workshop from Happy Hao Car.

• Turn Existing Website Traffic Into More Opportunities Through Personalization
• Why FAQ Content Matters More Than Ever for AI Search
• How to Measure AI Search Traffic
• Press Release: Happy Hao Car Launches OnPrompt
• Press Release: Happy Hao Car Unveils Sidekick
• Spring Cleaning Your VDPs
• Time to Spring Forward: 5 Quick Updates to Boost Engagement on Your SRP
• Start 2026 Strong: Dealer Marketing Lessons from 2025
• AI Search Visitor Behavior
• AIO and GEO Aren’t Replacing SEO

Kelcey Drapp is an SEO Manager working at Happy Hao Car since 2015, and in the digital space for nearly a decade. Kelcey enjoys digging deep into SEO research to discover the 'why,' and helping clients realize their potential for success online. In addition to her work in local SEO, Kelcey also likes to volunteer her digital marketing skills for non-profits in need. When Kelcey is not working, she enjoys cooking, traveling, and spending time with her husband, son, & three dogs.

Join the discussion 3 Comments

• Ravi Kumar says: September 3, 2019 at 6:03 am Thanks Kelcey Drapp, Sharing a great article, I have one issue regarding URL, I edited the old URL, but in Search Console, in the coverage section its showing Error due to 404, also showing URL in Index, I want to show my edited URL. I also deleted this URL in webmaster but still showing in the index, plz guide what can I do for removing forever. Another Question is I have deleted my old page in Search console in Remove section, still, show in Index, how can I delete forever? Kindly Guide me. Reply Kelcey Drapp says: September 3, 2019 at 5:56 pm Ravi, make sure you’re using 301 redirects to point old pages to new pages. You can also look into 410 status codes to makes those pages drop from the index for good; but if you’ve notated these changes in search console, you should see pretty quick results. Reply
• clarachristy says: October 18, 2019 at 12:41 am The information you’ve shared in blog is remarkable. Thanks for sharing such quality information. Reply

Save my name, email, and website in this browser for the next time I comment.

• Generative Engine Optimization (GEO)
• – Multi-Rooftop Groups We’re designed for groups – large or small. Discover how our products and solutions can help your marketing teams and agencies do more with less.
• Engage Customers & Generate Leads Experience outstanding lead generation with our powerful Cosmos platform. Switch to us for unmatched performance and customer engagement.
• – Generate More Website Traffic Boost sales with responsive websites. Experience qualified leads, top-notch SEO results, and seamless mobile experiences for dealership success.
• Automate & Increase Efficiency See how our Marketing Technology solutions can generate Thousands of dollars in savings for your dealership or dealer group.
• – Analyze Performance & ROI Analyze product and market performance with data you can trust. Powered by GA4.
• Meet Compliance Requirements Compliance solutions to keep your dealership or dealer group ahead of ever-changing privacy and accessibility laws.
• – Don’t See Your OEM? Happy Hao Car provides websites, SEO, and digital advertising to all vehicle OEMs. REACH OUT
• – Product Updates The latest digital marketing innovations driving extraordinary results for your dealership.
• – Happy Hao Car Webinars Get expert advice on how to get the most out of your dealership’s digital marketing efforts.
• – Read the Blog Explore our blog for news, tips, and insights. Discover articles providing valuable information to enhance your marketing strategies.
• – An Award Winning Company We offer an unwavering commitment to top-converting websites, premier automotive SEO, and exceptional digital marketing has earned us industry recognition.
• Leadership The leadership team is composed of digital and automotive experts with extensive experience. They drive our mission to serve the automotive industry with dedication.
• – Why We Are Different Experience the Happy Hao Car difference – increased car sales, exceptional service, and 99% customer satisfaction.
• Join the Team Join our dynamic team providing cutting-edge online marketing solutions to automotive dealerships across the Americas.
• – Testimonials Read about the great success our clients have experienced with Happy Hao Car.
• – Third Party Partners OnMarketplace creates a large ecosystem of vetted partner applications which can be added quickly and easily to a dealer’s website to maximize site impact and effectiveness.
• – Partner Standards Learn about the standards required along with the key business value delivered by Happy Hao Car and each of our trusted third-party partners.