Observability for Ruby's 48 sales agents
Ruby is an AI sales assistant for B2B revenue teams. It runs 48 agents as thousands of parallel instances: one per rep, per account, per deal. A rep points it at an account, a deal or a calendar, and Ruby does the research and writing a sales engineer would do before a meeting, then writes the results into CRMs such as Salesforce and HubSpot.That last step is why observability matters so much here. Ruby set itself one rule: any agent output that reaches a customer must be traceable to the exact prompt, evidence and cost that produced it. This is how it got there on Langfuse.
Why it was hard
Fan-out and nesting. A parent agent spawns sub-agents, model calls run in thread executors, and research tools run in parallel. Ambient tracing context does not follow all of that, and an orphaned model call shows up as a flat, unattributed row that explains nothing.
Blast radius. Every surface, from account research to campaign personalisation, is a place where a model can be wrong in a way that lands in a real CRM contact. The question is never whether something failed. It is which of dozens of steps went wrong, what the model was shown, and what it cost.
Customer data in traces. Traces carry prospect and rep information. Everything leaving the process has to be masked before it is sent, with exactly one deliberate exception, for attribution.
One run, one trace
Ruby's agents work across four surfaces: account research, deal preparation, an analyst chat and campaign personalisation. Anything that writes to the CRM is proposed on a card the rep confirms before it lands.
Every agent run becomes one Langfuse trace, and the trace id is derived from Ruby's own execution id. The database row and the trace are the same object seen from two sides. Deterministic quality scores run on every run, two LLM judges run on a sample, rep feedback lands on the exact run, and three alert rules page Slack within the hour. Prompts for all but two agents live in Langfuse, so a change or a rollback is a label move rather than a deploy.
Following is an example of one agent run as one trace:
Because the trace id is derived from our own run id, anything that knows the run can find or score the trace later with no mapping table. Our database row and the Langfuse trace are the same object seen from two sides.
Ali Zaman, Lead Engineer, Ruby
Rolled out one layer at a time
Tracing went live on Langfuse Cloud in April 2026, with full coverage and no sampling. The quality, prompt and alerting layers followed over the next six months, each added only once it had proved itself on production traffic. The order mattered:
Start with tracing only: every model call recorded, no sampling.
Derive the trace id from the run id before adding anything else.
Move prompts into Langfuse agent by agent, keeping an in-code fallback.
Add judges only after calibrating them on known-answer cases.
Fire every alert deliberately with seeded data before relying on it.
How Ruby uses Langfuse
Ruby runs on Langfuse Cloud through the Python SDK over OpenTelemetry. One project holds every environment (production, staging, demo, UAT and local) plus two reserved for experiments and for the judges' own model calls, so test runs and judge calls never mix with live scores. A handful of engineers run the whole fleet, and the same traces serve everyone: engineers debug from them, product reads cost by agent from them, and reps score answers into them from inside the product.
Tracing: every model call, typed, nested and priced
Each run is one trace with typed observations for agent, tool and generation, which is what draws the agent graph view. Nesting is explicit, through a Python context variable, so sub-agent calls land under the sub-agent even across thread pools. The session is the deal and the user is the rep.
The model provider returns token counts but nothing Langfuse can price natively, so each generation carries its own usage and cost, including cache-read and cache-creation tokens, from the same price table that feeds Ruby's own database. Tracing is best-effort by design: it can never fail an agent run. See Langfuse's docs on tracing and token and cost tracking.
The trace header: latency, session, user, environment, cost and tokens on one line. Demo environment.
Prompts: promotion is a label move, not a deploy
Eighty prompts live in Langfuse: ~50 agent and helper prompts and 26 quality rubrics, each named after its agent. Versions carry dev, staging and production labels, and moving a label puts a change live in about two minutes, through two 60-second caches.
Every generation is linked to the exact prompt version it used, and that version is also stored on the run's database row. "Did version 7 make the analyst better or worse?" is a filter, not an archaeology project. Each agent keeps an in-code fallback for when Langfuse is unreachable. See Langfuse's docs on prompt management.
Scores and judges: cheap checks on every run, expensive judges on a sample
Deterministic scores are computed in code on every run: confidence, output length and structure, and a flag that fires if raw tool markup leaks into a customer-facing answer.
Two managed evaluators run on production traffic through Ruby's own model key:
A grounding judge, on 20% of runs, scores against an anchored rubric in 0.2 steps and must list every claim as supported or unsupported before it scores.
A wrong-entity judge, on 10% of runs, checks that the output is about the prospect and not the seller.
Both were tested against known-answer cases with planted mistakes, measuring false alarms and misses separately, before any score was trusted. Rep thumbs land on the exact run as a score with an idempotent id. See Langfuse's docs on LLM-as-a-Judge.
Code and judge scores side by side on one run, with the judge's claim-by-claim comment. Demo environment.
Datasets and experiments: model changes are gated by golden sets
Golden datasets are built from reviewed production runs plus deliberately hard cases, such as failed research and thin context, each with a note on why it was included. A new model runs against the golden set beside the current one, both outputs are judged, and the comparison is reviewed before any configuration changes.
The move to Sonnet 5.5 and the grounding judge's switch to Haiku 5.5 were decided this way. The wrong-entity judge stayed on Sonnet 4.6, because the cheaper model raised a false alarm. See Langfuse's docs on datasets and experiments.
Dashboards and alerts: one-hour windows, and no data instead of zero
Each environment has one dashboard: cost by agent, cost by model, grounding and wrong-entity scores over time, and observation volume by level, where the ERROR line should stay at zero. Three alert rules post to Slack per environment: more than five ERROR steps in an hour, a grounding average below 0.4, and a wrong-entity average below 0.8.
Two settings mattered more than expected. Windows are one hour, because day-long windows evaluated too slowly to page anyone in time. And score monitors show no data rather than treating a quiet hour as a quality collapse. See Langfuse's docs on dashboards and alerts.
What it changed
Observability did not make our agents good. It made it possible to know when they were not, within an hour, with the exact prompt and evidence in front of us. That is what let us put agents in front of a customer's CRM: for any single run we can show what it saw, what it decided and what it cost. For a product like ours, that is the difference between a feature and a liability.
Raja Meer Baz, Head of Product, Ruby
Full coverage, no sampling. Every run of all 48 agents is traced and priced to the cent, across production, staging and experiments. Cost by agent, model and rep is a filter rather than a finance exercise.
Quality measured, not assumed. Grounding averages 0.84 across about 1,500 judged runs and wrong-entity 0.97 across about 280, with zero tool markup leaks in 340 scored customer-facing responses.
Problems found that nothing else would have caught. Typed observations surfaced an erroring tool within a day of going live. A judge that flagged sound research turned out to be missing the evidence the agent had used: once the trace carried it, the same run went from 0.2 to 0.95. Both led to one standing rule: if a model call is not in the trace, it does not exist.
Faster change, less risk. A prompt change ships in about two minutes instead of a deploy. Moving account research from a sequential chain to parallel levels cut a full run from roughly 35 minutes to about 11, measured in Langfuse latency. Every token is attributed before a feature is priced.