Walmart scrapers in production
Heyo, story time: Spent the last year running Walmart scrapers in production. Headless browsers (Playwright specifically) are almost always recommended over plain "requests" + BeautifulSoup for JS-heavy sites like Walmart, and that's true, but "use a headless browser" isn't the whole story. Here's what I learned that actually works in practice You may ask why depend on headless at all? Walmart's product pages are JavaScript-rendered. A raw HTTP request returns an HTML shell, prices, titles, and availability are injected by JS after load. BeautifulSoup never sees that data. Now for the headless browser part, it runs a Chromium engine, executes the JS, and lets you query the fully-rendered DOM. That part works well. Even with a headless browser, you'll hit blocks, it's not a holy grail as some of people have praised it over the reddit. Walmart fingerprints more than just your IP, browser canvas signatures, WebGL data, timing patterns, and TLS handshake characteristics are all signals. Vanilla Playwright out of the box is detectable. You need "playwright-stealth" or equivalent patches to mask the most obvious headless tells.
Walmart A/B tests constantly. The "<h1>" for the product title and "<span itemprop="price">`" for pricing, the selectors everyone uses, can and do shift. A scraper that worked Monday can silently return empty strings by Wednesday. You need selector fallbacks and output validation, not just "element.inner_text()". As for the resources, well, each Chromium instance eats ~150–300MB of RAM. If you're running concurrent scrapers, this adds up fast. For small datasets it's fine at scale, you either need careful concurrency limits or a distributed setup. Rotating proxies help with IP bans but don't solve fingerprinting. Worse, misconfigured proxies inside a browser context can cause silent failures, the request goes through but returns a CAPTCHA page that your parser doesn't catch. Always validate that your response actually contains product data before storing it.
Honest suggestions, people:
- ALWAYS USE "playwright-stealth" to patch headless fingerprints
- Add "wait_for_selector()" with a timeout before extracting, don't assume the element is there
- Build in retry logic with exponential backoff on failures
- VALIDATE YOUR OUTPUT: if price is empty string, treat it as a failed scrape and retry
- Rotate User-Agents per session, not per request
- Use residential proxies, not datacenter, Walmart's filters are tuned to spot datacenter ranges, (however, I was running datacenter at first with the help of residential proxies, ditched datacenter after some time).
Headless browsers are the right tool for Walmart, but they're not a reliability silver bullet as some of you praise it. For me particularly, ~85–90% success rate with a well-tuned setup was what I got at most, dropping toward 60–70% if you skip stealth patches and output validation. The remaining failures are mostly CAPTCHAs and transient blocks that retries will catch. For anything production-scale, budget time for maintenance. Walmart's defenses update, and your selectors will break. That's just the reality of scraping a site this sophisticated.