Building a copilot that says "I don't know" instead of guessing

The situation

Support and operations staff answered customer questions from a scattered mix of wikis, PDFs, old email threads and whatever the most experienced person in the room happened to remember. Two agents would give two different answers to the same question depending on which document they found first, and a new hire took months to reach the point of trusting their own answers.

The constraint

The obvious approach — point a retrieval system at the document store — runs straight into the actual problem: the documents disagree with each other. An old policy PDF and a newer wiki page say different things, and a naive system will retrieve whichever is a better semantic match, not whichever is current. Some documents are also access-restricted, and a copilot that leaks a restricted answer to the wrong user is worse than the inconsistency it was meant to fix. And it had to live inside the helpdesk tool agents already had open, not become one more tab nobody opens under deadline.

What we built

Citations, and a real refusal path

Every answer points to the specific document and passage it came from, and the model is explicitly allowed to say it does not know rather than produce a fluent guess. Refusal rate is a metric we track on purpose, not a failure to minimise to zero — a wrong confident answer costs more than an honest no.

Freshness as a first-class property of a document

Every source carries version and effective-date metadata, and documents can be marked as superseding an older one. Once superseded, the old document stops being retrieved, closing the exact failure mode that started the project: two agents, two answers, both technically "in the knowledge base."

Permission-aware retrieval, not permission-aware display

The index is filtered by the querying user's access group before retrieval happens, not after. A restricted document is invisible to the model for that query, not merely hidden in the response — the same principle we apply to any retrieval system handling documents with mixed sensitivity.

Embedded where the work already happens

The copilot lives inside the helpdesk tool as a sidebar widget rather than a separate destination. Adoption depended entirely on this; a second tab is a second thing to remember to check.

A feedback loop that fixes the source, not just the answer

Agents can flag an answer as wrong directly in the widget. Flags route to the document owner, not into a model-tuning backlog — because the fastest way to fix a wrong retrieval is usually to fix the document that was wrong, not the model that faithfully retrieved it.

What changed

Answer consistency across agents improved, and new hires reached productive competency in days rather than weeks. The unplanned benefit was the flag queue itself: it became a running list of exactly which documents were stale or contradictory, which nobody had a way to see before.

What we would do differently

We found far more duplicate and conflicting documents than expected once ingestion actually ran. The document-freshness and superseding workflow should have shipped before the broad rollout, not been added in month three after agents started noticing two different answers to the same question — by then the trust cost of the inconsistency was harder to undo than the engineering cost of building it earlier would have been.

Outcomes

First-response answers needing escalation

cut substantially

Time to productive competency, new hires

weeks → days

Stale or conflicting source documents surfaced and fixed

dozens in the first quarter

Client identity withheld under a mutual NDA. Figures are illustrative — rounded and directional, meant to show the shape of the change rather than an audited result. We will walk through the real numbers, and how they were measured, under NDA on a call.