u/Relative_Recording47

I’m facing a crawl budget issue on a large WordPress site (20K+ pages). Thousands of feed URLs are being crawled repeatedly by Google, even though I’ve already taken multiple steps to block and handle them.

Almost every blog and landing page has an associated /feed/ URL, and Google is crawling ~15K+ of these every 2–3 days. Some of these feed URLs are even getting indexed, which is something I want to completely eliminate.

Here’s what I’ve implemented so far:

• Blocked feeds via WordPress settings/plugins
• Added rules in robots.txt:Disallow: /feed Disallow: */feed/ Disallow: /?*
• Implemented conditional handling:
  • For normal users → feed URLs are 301 redirected to the canonical (non-feed) version
  • For bots → feed URLs return 410 (Gone)

Despite all this, Google continues to crawl a large volume of feed URLs consistently, which is negatively impacting my crawl efficiency for important pages.

What I need help with:

• How can I fully stop Google from crawling these feed URLs?
• Is there a better approach than 410 for bots + redirect for users?
• Any proven strategies to reclaim crawl budget from faceted/feed URLs at scale?