u/SaaS2Agent

A lot of healthcare AI products look great in demos.

The assistant answers well, collects intake details, summarizes the patient’s concern, and maybe routes them to the next step.

But honestly, I think the hard part starts after the demo.

In healthcare, the real question is not just “did the AI give a good answer?”

It is more like:

- What patient data did it actually see?

- Was that data even allowed to enter the model at that point?

- Did safety checks run before the agent took action?

- Could it call a tool too early?

- Did it stay within its role, or slowly drift into clinical advice?

- Can someone replay the exact interaction later and understand why it behaved that way?

- And when the system should stop, who owns the handoff?

The more we work around healthcare agents, the more I feel the agent itself is only one part of the product.

The real product is the governed workflow around it: PHI boundaries, role limits, safety gates, context control, tool permissions, replay, QA, and human review.

A chatbot that sounds good is very different from a healthcare AI system that is actually safe to release.

For people building or working in healthtech, where do you usually see things break first: compliance, clinical trust, workflow design, or production QA?

One thing I keep running into with AI agents is that testing the prompt is only a small part of the problem.

An agent can give a decent response in a simple test and still break once it has to move through a real workflow.

The weird failures usually show up when it has to:

• remember context across steps
• pick the right tool
• recover from a failed tool call
• decide when to ask the user for clarification
• pause for approval
• avoid repeating the same action
• handle inputs that are vague or incomplete

That feels very different from testing a chatbot response.

For a normal SaaS feature, we can usually define the expected flow pretty clearly.

For an agent, there are many possible paths, and some of them only appear when users behave unpredictably.

I’m starting to think agent QA needs to be closer to scenario testing plus behavior checks, not just evals on final answers.

I ended up turning these failure patterns into a small QA checklist for agentic workflows, mostly for my own use. Not sure if others are dealing with the same thing, but happy to share it if useful.

How are people here testing agents before putting them in front of users?

Are you mostly doing manual test chats, scripted scenarios, trace review, synthetic users, or something else?

I’ll go first: “useful” means something different now.

A few years ago, a clean dashboard and a solid workflow could pass.

Users have started expecting the product to do more of the actual work, probably because AI agents and copilots have changed what “software” feels like.

Support tools are the easiest example. Showing tickets is not enough anymore. People expect summaries, reply drafts, routing, follow-ups, pattern detection, all that stuff.

Same thing is happening in CRMs, analytics, HR, finance, ops tools, pretty much anywhere people are stuck doing repetitive work.

“Here’s the data, now go figure it out” feels a lot weaker than it used to.

What’s the hardest shift you’re seeing in your product?