u/Paulinefoster

After digging through the Reddit vs Perplexity lawsuit, I found a pretty interesting example.

To prove that Perplexity was indirectly getting Reddit data through Google, Reddit set up what they called a “honeypot test,” basically the digital version of marked bills.

The idea was simple: Reddit created test posts that could only be accessed by Google’s search crawler. A few hours after those pages were indexed by Google, the content from those test posts started appearing inside Perplexity answer queries.

So Perplexity apparently was not building its own search engine for this. Instead, it was buying services from third-party providers and indirectly getting Reddit data that had already been crawled by Google.

My guess is that a lot of other AI companies work similarly too. They mostly rely on Google’s data layer to build answers.

The original filing mentions this on page 27.

Lately there’s a lot of discussion around llms.txt, with some calling it the new robots.txt for AI crawlers.

Some say adding it helps LLMs better understand website content and improves visibility in tools like ChatGPT, Perplexity, or Gemini. Others think it’s mostly hype and not something that makes a measurable difference right now.

Has anyone actually tested this and seen real impact? Curious whether adding llms.txt is becoming important for AI visibility or if it’s still too early to matter.

According to Google:

AI Overviews: AI Overviews are a normal search with a few fan-out queries. It triggers a more complex and longer query, then the AI Overview combines everything into a summary. With AI Overviews, the whole retrieval and ranking system is still the old style, and the AI Overview feature sits on top of it and operates with its own AI.

AI Mode: AI Mode works with longer queries and longer conversations, and users like the conversational aspect. AI Mode does use Search, it uses fan-out queries, and it has linked results and citations. But it is its own thing because the infrastructure is new. It runs on Search, but it has its own bigger platform.

I tested it myself. AI Overviews seemed to pull from only the first few links, while AI Mode pulled from many more links.

So it is pretty normal if your page ranks on page one but is not pulled into AI Overviews. There is no need to be anxious about it. I believe users will still browse your page.