A governed retrieval system that answers employee questions from internal docs — with permissions, citations, and an eval harness that catches drift.
Every scaling SaaS company hits the same wall: the answer exists, but it is scattered across Notion, a dozen Slack channels, Linear tickets, GitHub READMEs, and a Zendesk macro nobody maintains. A new engineer loses a week finding what a senior engineer knows cold; support pings the same three people for the same edge cases. A naive chat-with-your-docs demo is trivial to build and dangerous to ship — it will happily answer from a stale RFC or leak a doc the asker should not see. An enterprise-grade internal retrieval system is a different animal: it respects source permissions so answers never cross access boundaries, it cites every claim back to the document and section so people can verify, and it ships with an eval harness that catches accuracy drift as your docs change. The point is not a clever chatbot — it is a governed answer layer the whole company can trust.
A 500-person SaaS company indexed Notion, Confluence, and resolved Zendesk tickets behind their existing SSO groups; engineers and support each see only what they are entitled to, and every answer links its sources. New-hire ramp time to first independent on-call shift dropped noticeably because the assistant answers the where-is-this questions a buddy used to field. Their platform team runs a weekly eval set of 80 golden questions against the index and gates any retrieval-config change on it, so a re-chunking change cannot silently degrade answer quality. The support org saw repeat internal escalations to senior engineers fall as tier-1 self-served from cited internal answers.
Catalogue every knowledge source, its owner, freshness, and sensitivity. Decide what is in scope — and explicitly exclude what is stale or unowned.
Index documents with their access-control metadata so retrieval filters to what the asking user is already entitled to see. Permissions are a retrieval-time filter, not an afterthought.
Every answer links the exact passages it used. No citation, no answer — the model returns 'not found' rather than guessing.
Curate a golden set of 100-300 real questions with verified answers. Score retrieval recall and answer faithfulness on every change, in CI.
Launch to one team, capture thumbs-down with reasons, fix the underlying docs or retrieval, then expand. Treat bad answers as documentation bugs.
Tuned for SaaS & Tech Companies. Use as-is or adapt to your voice.
You answer employee questions strictly from the retrieved internal documents provided. Rules: (1) Use ONLY the supplied passages; if they do not contain the answer, say so and suggest who or which space likely owns it — do not answer from general knowledge. (2) Cite every factual sentence with its source title and section. (3) If two sources conflict, surface both and note the more recent one. (4) Never reveal content from a passage the requester’s role is not permitted to see (permission metadata is attached to each passage). (5) Prefer a short, correct, cited answer over a long, confident, unsourced one.
For each golden question, score the system 0–2 on each axis: Correctness (2 = factually right and complete), Grounding (2 = every claim cited to a real supplied passage; 0 if any hallucinated cite), Permission-safety (2 = no leakage across roles; an automatic FAIL of the whole run if violated once), Refusal-quality (2 = correctly says I don’t know when the answer is absent). Track the run score over time; block any index/chunking/prompt change that drops aggregate Grounding or any Permission-safety failure.
Before indexing a source, record: owning team; SSO group(s) that may read it; sensitivity tier (public-internal / confidential / restricted); freshness expectation (is stale dangerous?); and a deprecation owner. At query time, filter passages to the requester’s groups BEFORE ranking, never after. Re-sync permissions on the same cadence as the source system. Audit quarterly: pick 20 restricted docs and confirm no unauthorized role can surface them via the assistant.
{"question":"...","expected_answer":"...","must_cite":["doc-id-1"],"acceptable_to_say_unknown":false,"owner":"team"}For each generated answer: extract every claim, verify each is supported by a cited passage. Faithfulness = supported claims / total claims. Block deploys that drop below 0.95 on the golden set.
Get one new AI workflow per week, tuned for SaaS & Tech Companies teams. Real templates, real ROI.
Skip this if your underlying documentation is wrong or contradictory — RAG will faithfully retrieve the wrong answer. It is a retrieval system, not a fact-checker. Fix document ownership and freshness first.
One practical AI tip per week. No fluff.
Get the full guide with niche templates and workflow imports.