AI Agents are basically silent crawlers at this point
AI agents are already hitting sites and most people have very little visibility into it.
For example, Claude browses your MCP docs, GPT scrapes pages, OpenClaw creates shitposts on Reddit, etc.
I've been working with the team at Vouched recently on a way to detect and identify agents, bots, human traffic, etc. to help with this. Under the hood, there is an ecosystem of agents on a registrar with appropriate reputation scoring, all powered by a identity layer named KYA-OS which uses verifiable credentials that persist identity / authentication context across systems.
Well in the world of vibe coding we wanted it to be accessible to non-technical people too so it's as simple as:
- copy prompt from docs
- paste into Claude Code
- merge your PR
The hope is that people can have better visibility into which agents (or bots, or people) are interacting with your site and what they're actually doing.
If you're interested, I'll share the docs in comments.