Glossary
Model monitoring
Model monitoring is watching what a live system actually produces over time, so a decline in quality is noticed from evidence rather than from somebody complaining.
In plain terms
Keeping an eye on what the thing is producing now, rather than on what it produced during testing. Most software monitoring answers whether the service is up and how fast it responded. Both can be perfect while the answers themselves quietly get worse, and noticing that is a different job needing a different measurement.
Why it matters
Because the alternative is finding out from a customer. Quality decline here is silent: nothing errors, nothing slows down, and the system reports itself healthy throughout. Without something watching the output, the first signal is a complaint, and by then the same problem has usually been reaching people for weeks.
How it works
It measures the output rather than the service, which is what separates it from ordinary monitoring. Availability and response time are properties of the system; whether the answer was any good is a property of the content. A dashboard full of green can sit alongside a steady decline in usefulness, and nothing on that dashboard is wrong.
Something has to produce the judgement, and there are only a few options. A person reviewing a sample, a model scoring output at volume, or a proxy signal such as how often people rephrase, escalate or abandon. Each is imperfect: the sample is small, the model needs its own validation, and the proxy measures frustration rather than quality. Most working setups use more than one.
It is more useful as a comparison than as an absolute, because absolute quality is difficult to define and easy to argue about. A number that is stable and then moves tells you something actionable regardless of what it means in isolation. That is why the first genuinely useful thing to record is usually whatever you can measure consistently, rather than whatever would be ideal.
The measurement has to survive the system changing underneath it, which is the part most often overlooked. If the model behind a product is updated, or the material it reads is refreshed, a series spanning that change is comparing two different things. Recording what was in place alongside the number is what keeps the history readable a year later.
Two dashboards, one healthy system
Seen in the wild
A platform with one surface watching conversation quality and another running tests of how the agent behaves, which is this practice sold as a product feature.
SierraTracing and evaluation tooling that turned debugging agent behaviour into a discipline rather than archaeology, which is the same job for a team building rather than buying.
LangChain / LangGraphBringing your own model keys and watching usage directly, so the economics stay visible while autonomy grows.
Relevance AI
Common misconceptions
People assume
Our observability already covers this.
In fact
It covers whether the service answered and how quickly, which is genuinely necessary and says nothing about whether the answer was right. The failure this addresses produces no error, no latency spike and no alert, so a system can be entirely healthy by every existing measure while getting worse.
People assume
It is only worth doing once quality problems appear.
In fact
By then there is nothing to compare against. The value is almost entirely in the baseline: a number recorded while things were fine is what makes a later movement legible. Starting after a complaint means knowing something is wrong and not being able to say when it started or how far it has gone.
Telling them apart
Model monitoring vs Drift detection
Model monitoring
Watching output quality over time, whatever the cause.
Noticing one particular cause: the inputs or the world moved.
Drift is one of the things monitoring can find. Monitoring is how you find it, and it will also find several other things.
Questions
- What is worth recording first?
- Whatever can be measured consistently without much effort, because consistency matters more than sophistication at the start. A crude number recorded every week beats an excellent one recorded twice, since the whole value comes from the comparison and a series with gaps in it cannot support one.
- How does this relate to using a model to score output?
- That is one instrument this practice can use, and the most common one at volume. It is not the practice itself, and it needs its own validation before its numbers mean anything, which is worth keeping separate in the reasoning as well as in the terminology.
- Who should be watching?
- Whoever owns the system, since a measurement nobody is accountable for stops being read within a few months. The common failure is not that nothing is recorded but that a dashboard exists which no named person is expected to look at or act on.
Key takeaways
- Measures the output, not the service; uptime monitoring cannot see this failure.
- The judgement comes from a person, a model or a proxy signal, and all three are imperfect.
- More useful as a comparison than as an absolute, so the baseline is the valuable part.
- Record what was in place alongside the number, or the history stops being readable.
Tools that use this
- Sierra
Conversation quality watched and agent behaviour tested as product surfaces.
- LangChain / LangGraph
Tracing and evaluation tooling for teams building rather than buying.
- Relevance AI
Usage watched directly so economics stay visible as autonomy grows.
Last checked July 2026