Enterprise RAG: Good answers are not enough

The first success moment in a RAG prototype arrives quickly: documents are indexed, a question is asked, and the model returns a fluent answer with citations. That is impressive in a demo, but it leaves the hardest enterprise questions unanswered.

Is the requester allowed to see those documents? Is the retrieved evidence sufficient? Were the cited sources actually retrieved? And how can a technical change be shown to improve quality rather than silently degrade it? Those questions shaped my local Enterprise Knowledge Assistant.

The project uses a synthetic knowledge base only. It is neither a customer system nor a production deployment. That explicit boundary made it possible to explore architecture and quality questions without exposing confidential information.

The central design decision was not to let the model control citation integrity. Documents are chunked, embedded with multilingual E5, and retrieved through Chroma or pgvector. Allowed citations come from retrieval metadata. If the model names another source, a deterministic citation gate rejects the answer. Plausibility does not replace verification.

In an enterprise setting, hiding documents after an answer is too late. Unauthorized content must never enter re-ranking, prompts, or debug output. ACL and tenant filters therefore run before model context is assembled.

The demonstrator covers several maturity steps: freely supplied group headers for early experiments, server-side API-key-to-group mapping, and optional validated OIDC/JWT tokens. It is not a finished identity platform, but it proves the key rule: identity and authorization are verified outside the model and translated into retrieval filters.

LangGraph turns the flow into explicit steps: retrieval, evidence gate, context-grounded answer, and citation gate. If evidence is insufficient, the model is not called. If citations are invalid, the answer is not delivered.

That structure marks the difference between a loose prompt chain and a controllable quality process. Decisions become visible, independently testable, and traceable in debug mode. FastAPI, a local web interface, and request IDs add usability and operational context, while questions, document content, and retrieved context are deliberately excluded from request logs.

Chroma is useful for a fast local start. pgvector becomes attractive when vector search should move closer to existing Postgres operations. A working integration alone, however, says nothing about stable result quality.

Both backends were therefore compared with the same knowledge base, dataset, and metrics. Hit rate and MRR were equal in the documented run, while pgvector scored slightly higher on citation accuracy and keyword recall. Still, both live reports missed the strict quality thresholds, especially for abstention on unanswerable questions.

The most valuable result is not that a local model can answer questions about documents. It is the connection between requirements engineering, QA, and architecture: what claims may the system make, what evidence is required, who may use which context, and which metric decides release readiness?

Enterprise RAG becomes reliable when those rules become system behavior. Good answers still matter, but trust comes from verifiable sources, controlled access, visible limits, and evaluation that is allowed to produce a negative result.

If you are building a RAG or agent product and want to structure quality goals, guardrails, and delivery readiness early, let us make the critical decisions visible before rollout.