u/New-Reception46

Quick backstory like some of you mentioned in those hiring rants. I handle backend and some ops for our team, been at it 5 years. We rely on these SaaS dashboards and admin panels for tracking everything, but half the time key actions like bulk updates or exports arent exposed via API. Its either missing or so limited you hit walls fast.

Last week I spent 3 hours manually clicking through an internal tool to reset user sessions because no API endpoint for it. MFA everywhere makes scripting impossible without hacks. At the same time, managers are pushing harder for automation and efficiency, but without proper backend access it feels like being told to optimize something youre not allowed to touch. We can automate everything on paper, but the moment a workflow depends on UI only actions, it becomes a human bottleneck again.

Heard whispers of browser automation tools or AI agents that mimic human clicks, stealth scraping stuff that handles anti bot measures. But not sure if thats overkill.

If u was on my spot would you just accept the manual grind or got tools that bridge this gap?

Mid-sized org, around 300 people. Running multiple Databricks workspaces across AWS and Azure, hundreds of Spark jobs daily.

Debugging slow jobs, skew, small files, memory spills, shuffle issues takes too much manual time. Spark UI and Databricks monitoring cover the basics but at this scale someone always has to notice something is wrong before investigation starts. That lag is becoming a real problem.
Started looking into Spark agents to automate detection and surface bottlenecks without someone manually going through every stage. The volume justifies it but not sure what's production-ready vs still experimental.

What are people using for Spark agents at this scale? Does it hold up across multiple workspaces or does it fall apart once jobs get into the hundreds daily?