u/AdditionalWeb107

Hey peeps - I think the hardest thing about building agents is their evaluations. especially for scenarios that require multiple tool calls and the agent itself can go down a trajectory that you haven't manually tested before. And trajectories are voluminous and non-deterministic, and reviewing each one, whether through human review or auxiliary LLMs, is slow and cost-prohibitive.

So I built a signal-based framework for triaging agentic interaction trajectories. My approach computes cheap, broadly applicable signals from live interactions and attaches them as structured attributes for trajectory triage using OTEL attributes

I organize signals into a coarse-grained taxonomy spanning interaction (misalignment, stagnation, disengagement, satisfaction), execution (failure, loop), and environment (exhaustion), designed for computation without model calls.

In a controlled annotation study on τ-bench, a widely used benchmark for tool-augmented agent evaluation, we can show that signal-based sampling achieves an 82% informativeness rate compared to 74% for heuristic filtering and 54% for random sampling, with a 1.52x efficiency gain per informative trajectory.

The advantage is robust across reward strata and task domains, confirming that signals provide genuine per-trajectory informativeness gains rather than merely oversampling obvious failures. These results show that lightweight signals can serve as practical sampling infrastructure for agentic systems, and suggest a path toward preference data construction and post-deployment optimization.

Links to the approach and the project where this is implemented below

Hello Peeps Salman, Shuguang and Adil here from Katanemo Labs (a DigitalOcean company).

Wanted to introduce our latest research on agentic systems called Signals. If you've been building agents, you've probably noticed that there are far too many agent traces/trajectories to review one by one, and using humans or extra LLM calls to inspect all of them gets expensive really fast. The paper proposes a lightweight way to compute structured “signals” from live agent interactions so you can surface the trajectories most worth looking at, without changing the agent’s online behavior. Computing Signals doesn't require a GPU.

Signals are grouped into a simple taxonomy across interaction, execution, and environment patterns, including things like misalignment, stagnation, disengagement, failure, looping, and exhaustion. In an annotation study on τ-bench, signal-based sampling reached an 82% informativeness rate versus 54% for random sampling, which translated to a 1.52x efficiency gain per informative trajectory.

Links in the comments below