u/BeneficialLook6678

we’re building out some AI features for our app, things like chat responses and recommendations. mostly using gpt4o with some fine-tuning, expecting around 10k users once it’s live.

rn we rely on basic output tests and some manual reviews, but it’s slow and doesn’t cover edge cases well.

we tried adding tracing and eval tooling, but setup and maintenance ended up taking more time than expected. integration into our workflow has been the bigger issue than the tools themselves.

pressure from product to move faster, but our last beta surfaced a few hallucinations that almost made it to production. trying to find a way to validate behavior more consistently without turning it into a full-time effort.

what approaches have worked for you in catching issues early without slowing things down too much?

We keep running into the same problem as service systems get more automated
anytime something in how agents behave needs to change, it’s not really a small update anymore even things like routing logic, escalation rules, or how an agent responds in certain cases usually end up as dev work, not something ops teams can just tweak.

the frustrating part is that the people closest to the workflow already know what needs to change, but they’re stuck waiting in a development queue for it to happen.
this is what ends up happening in real workflows:

- support teams spotting repeated ticket patterns but not being able to adjust how those tickets get handled
- ops noticing escalation delays but needing engineering to modify the flowsmall process fixes sitting in backlog because they’re “not urgent enough” for dev cycles
- service managers relying on workarounds instead of directly updating agent behavior
- every improvement turning into a request instead of a quick adjustment

the direction things are moving toward is giving that control back to the people running the service, so changes to agent behavior and workflows can be made directly as things evolve, without turning every adjustment into a development task.

how are teams handling this in real environments without slowing everything down or depending on engineering for every change?

Genuinely uncomfortable situation and I'm not sure what the right call is from a purely technical standpoint.

One of our employees passed away unexpectedly about two weeks ago. Family notified HR directly. HR notified IT. We went to disable the account in Entra and deprovision from Okta the same way we would any termination, and HR stopped us. Their position is that until legal formally processes the separation, they can't update the HRIS status, and therefore IT shouldn't take any action that might interfere with estate or beneficiary processes.

Legal wants a certified copy of the death certificate before they do anything. The family is dealing with everything you'd expect them to be dealing with and hasn't submitted documentation yet.

So right now we have an active account, valid credentials that presumably no one knows except the individual who is no longer here, sitting fully provisioned with access to all the same apps and data as before. No one has logged in since the day before they passed — we can see that in the sign-in logs — but the account is technically open.

Our security team is pushing us to at minimum force a password reset and revoke all sessions. HR says that's still "account action" and they want to hold everything until legal clears it.

I get that there are processes for a reason but I'm struggling to understand what the actual risk of a session revoke is to any estate or benefits process. Has anyone been through this? Is there a documented approach for handling this gap between "we know the person is gone" and "we have paperwork to prove it"? Specifically wondering if others have gotten legal to agree on a middle ground — like read-only preservation mode or something — while the formal process catches up.