u/Agreeable_Bit5683

Context

Been rebuilding a small directory‑style programmatic project on weekends. Last push added ~800 location pages in one batch. I noticed discovery was extremely uneven when relying on sitemap updates alone.

Problem

After publishing the batch, GSC started showing the usual pattern:

· majority stuck in "Discovered - currently not indexed"

· a handful moving to crawl

· indexing waves happening very late

For the sitemap‑only set, the first meaningful indexing wave showed up around day 10-18. Manual "Request indexing" obviously doesn’t scale when the UI caps you around ~50/day.

What I tested

Split new URLs into three discovery paths:

1. sitemap only

2. manual GSC requests (sample ~40 URLs)

3. automated submission via Google Indexing API + IndexNow

The idea was not ranking impact, only discovery / crawl timing.

Setup

For the API batch I didn’t bother writing my own queue + retry worker for submissions. I used a hosted indexing worker during the test https://indexerhub.com/ mainly so I could push batches and capture retry behaviour without writing infrastructure. The basic flow was:

· watch sitemap for new URLs

· push URLs to Google Indexing API

· push same URLs to IndexNow

· retry failures when API quota hit

Any queue/worker could do this, the point was testing the submission channel vs passive discovery.

Observed results

Rough numbers from the first two weeks:

· sitemap‑only batch: first indexing wave ~10-18 days

· manual GSC sample: some indexed in ~4-6 days

· API + IndexNow batch: first indexed pages appearing ~24-72 hours

About ~35-45% of the API batch showed indexed within the first few days.

Example from logs:

· API submission: May 3 14:12

· Googlebot crawl: May 4 02:33

Obviously not causal proof, but the crawl timing difference was pretty consistent.

Interesting findings

A couple things stood out:

· Bing picked up URLs extremely quickly once IndexNow fired

· Google still filtered heavily but crawled noticeably sooner when pinged

· retry logic mattered more than expected, ~9% of submissions initially failed due to quota

Takeaway

For small sites this probably doesn’t matter. But once you publish hundreds of URLs at once, discovery lag becomes the real bottleneck, not content production.

Curious how others here are handling indexing ops for larger batches.

Are people running their own Indexing API pipelines, or mostly relying on crawl discovery + logs?