Employee Knowledge Search

The exploration

“Chat with your company’s documents” is a weekend demo and a multi-year engineering problem wearing the same t-shirt. The demo retrieves some chunks and generates a fluent answer. The production system must answer a harder set of questions: was the asker allowed to see every word that shaped this answer? Is the answer built on the current version of the policy or the one replaced last Tuesday? And how would anyone know if answer quality regressed on Thursday? This exploration takes the position that those three questions — permissions, freshness, and evaluation — are the product, and the language model is a replaceable component in the middle.

The problem, concretely

• Company knowledge is scattered across wikis, drives, tickets, and chat, with search tools that rank by keyword luck
• New employees ask questions that were answered in writing three times already, in three places, with three different answers of three different ages
• Naive internal RAG deployments create a new problem: confident answers assembled from stale or unauthorized fragments, delivered with more authority than the sources deserve

Permission-aware retrieval, or nothing

The non-negotiable invariant: the system must never surface content the asker cannot open at this moment — not in an answer, not in a snippet, not in a “related documents” hint, not leaked through an embedding neighbor. That rules out the tempting shortcut of one shared index with post-hoc filtering bolted on. The exploration’s approach:

1. Access control lists are ingested alongside content and treated as data with the same freshness requirements as the text itself
2. Retrieval filters by the asker’s effective permissions at query time, before ranking — a revoked permission takes effect at the next query, not the next re-index
3. Generation sees only chunks that passed the filter; there is no path from the index to the prompt that bypasses it
4. Red-team probes for permission leakage run continuously in the evaluation harness, because this invariant is exactly the kind that silently rots

A useful side effect: “no results you can access” is itself a truthful, actionable answer, and the system says it plainly rather than padding around the gap.

Index freshness as a contract

A stale answer about a security policy is worse than no answer. Rather than promising “real-time,” the exploration defines freshness as a published, per-source contract: chat and tickets within minutes, wikis within an hour, file shares within a day — whatever the connectors honestly sustain. Every answer displays the age of its oldest contributing source. Supersession is modeled explicitly, so a document marked as replaced drags its successor into the results instead of competing with it. The measure that matters is not indexing latency in the abstract but stale-answer rate: how often did the system answer from something a human would call outdated.

Answers versus links

Generated answers and ranked links are not rivals; they suit different questions. Factual lookups with one authoritative source deserve a short cited answer. Ambiguous, contested, or judgment-loaded questions (“what’s our remote work policy for contractors?”) often deserve the three relevant documents with one orienting sentence — because a synthesized answer would have to silently resolve a conflict the organization itself hasn’t resolved. The exploration builds both response modes and a router between them, and treats the router’s choice as a first-class evaluation target. The fallback direction is fixed: when grounding confidence is low, degrade from answer to links, never the reverse.

Query pipeline

1. Question arrives web console or Slack/Teams
2. Permission resolution asker's effective access, resolved now
3. Filtered retrieval hybrid search over authorized chunks only
4. Mode decision cited answer, or links with orientation
5. Response with provenance citations, source ages, confidence
6. Human verifies citation clicks close the loop

The evaluation harness is the product

The central claim of this exploration: an internal search system without an evaluation harness is not maintainable, because every change — a new connector, a re-chunking, a model swap, a prompt tweak — silently moves answer quality in unknown directions. The harness is built before the pipeline is tuned:

• A golden-question set per deployment: real questions with known-correct sources and reviewed reference answers, including deliberately unanswerable questions and permission-trap questions that must return nothing
• Every pipeline change runs the full set; retrieval hit rate, citation precision, refusal correctness, and permission-leak count are tracked per change, per connector
• Production telemetry feeds the set: questions with poor feedback or zero citation clicks become candidate golden questions after human review
• Observability down to the retrieval trace — for any answer, which chunks were considered, which passed the filter, which were cited — so “why did it say that?” has a factual answer

Cost and latency engineering

Internal tools live or die on unit economics nobody celebrates. The working budgets: under 3 seconds to first token for an answer, under 1 second for a links-mode response, and a per-query cost low enough that nobody meters usage internally — the moment employees ration questions, the tool has failed. Engineering follows the budgets: embedding and retrieval on efficient infrastructure with generation reserved for queries that need it, aggressive caching of repeated questions (with permission context as part of the cache key — a subtle place to leak), and small models for routing and query rewriting with the larger model spent only on final synthesis.

System overview

• Connectors: wiki, document store, ticket system, chat archive — each shipping content, ACLs, and deletion events on its freshness contract
• Index: hybrid lexical-plus-vector store, chunked with document structure preserved; supersession and recency features feed ranking
• Serving: permission resolver, retriever, mode router, generator with citation enforcement — every emitted claim must bind to a retrieved span, or the claim is dropped
• Harness and telemetry as first-class services beside the pipeline, not scripts beside it

Data and privacy posture

Internal documents include HR matters, legal drafts, and payroll adjacency. The exploration’s stance: the index inherits the source systems’ permissions rather than flattening them, queries are logged for evaluation with employee identity separated from question text under a defined retention window, no customer or employee content trains any model, and works councils and security teams get the retrieval-trace observability too — auditability is not only for engineers.

Prototype scope

Two connectors (wiki and chat archive), one pilot team’s corpus, both response modes, and the harness with an initial golden set of one hundred questions. Deliberately excluded until the invariants prove out: writing back to sources, multi-team rollout, and any personalization beyond permissions.

Risks and open questions

• ACL fidelity varies wildly by source system; where ACLs are unreliable, the honest choice is to exclude the source, which may gut perceived coverage
• Golden sets drift as the company changes; harness maintenance is a permanent cost that must be owned by someone, or the whole thesis collapses
• Chat is simultaneously the freshest and least authoritative source; ranking it honestly against documentation is unsolved here
• Adoption gravity: if answers are good, people stop opening sources — which makes citation-click telemetry, the main trust signal, fade precisely as trust grows

Delivery phases

1. Connector and index spike with permission model proven against red-team probes
2. Harness and golden set before tuning; both response modes behind the router
3. Pilot with one team; stale-answer rate and refusal correctness measured against contracts
4. Additional connectors only as their ACL fidelity earns inclusion