u/Mesmer7

As the title says, My visitor logs show OAI-SearchBot/1.3 recieving inconsistent results when crawling my robots.txt file. Most of the time, it receives a 200 status, But sometimes it receives a 401, and other times it receives a 403.

I've whitelisted all of the IP addresses (in All-in-one Security) that OAI-Search is crawling from, but it made no difference.

I asked my hosting service for help. They restarted the server level firewall. But it made no difference.

Anyone else seeing this in their visitor logs? Any idea what causes it or what we can do about it?

Google Search Console has been reporting long response times for crawl requests on site for months, sometimes as long as 4 seconds. And I can't figure it out the cause.

• I'm on a litespeed server,
• I cleaned up the htaccess file. (eliminated the plugin that injects security headers, cleaned up the rewrite directives, etc.)
• Disabled the php firewall,
• I have LiteSpeed Cache installed (and well configured).
• I'm even priming the cache updates.

For a couple of weeks I had the response time down to 70 ms, but then yesterday it jumped back up to 3.5 seconds. And today it's back to 60 ms.

Can someone help me pinpoint the cause?

www. hypnoticdreams. com (NSFW)