Skip to content
Back to blog
Jul 20, 20268 min read

Testing AI systems: Why one test level is not enough

A practical, layered approach to quality assurance for LLM, RAG, and agentic systems—from the software foundation to domain-level answer quality.

AI TestingLLM EvaluationRAGQA Strategy

AI quality does not start with the answer

A fluent answer is not evidence of quality. An AI system is more than a model: it includes software architecture, data flows, permissions, prompts, knowledge sources, and operating processes. Every layer can create failures, even when the visible output initially sounds convincing.

That is why I test AI systems in layers. The conventional testing pyramid remains essential, but it is complemented by domain evaluation of model outputs. This makes it possible to verify both technical reliability and whether results are acceptable in their intended context.

The software foundation remains indispensable

I start with the conventional architecture: unit tests for deterministic logic such as validation, filters, parsers, and policy checks; integration tests for APIs, databases, model and search interfaces; and end-to-end tests for critical user flows. Security and performance tests are added so authentication, rate limits, error paths, and load behaviour are not discovered only in production.

That may sound standard, but it matters especially for AI applications. A model can generate a good answer while the application must still not deliver it—for example because an authorization filter fails, structured output is invalid, or a timeout leaves a process incomplete. These risks need deterministic tests.

Golden datasets make answer quality measurable

I also create a representative golden dataset: curated inputs covering real cases, edge cases, ambiguity, and expected refusals. Depending on the product, it contains reference answers, allowed facts, required sources, or clearly defined evaluation criteria. The key is not maximum size, but traceable coverage of relevant risks.

I use it to assess domain correctness, relevance, completeness, hallucinations, and format compliance. Not every dimension has the same weight in every use case. For a summary, completeness may matter more than exact wording; for a decision aid, source grounding and safe escalation under uncertainty are often decisive.

Evaluate retrieval and answers separately in RAG

For a RAG system, I deliberately separate retrieval quality from answer quality. First, I check whether the right documents or passages were found—for example using recall, hit rate, MRR, or a domain-specific relevance judgement. Only then do I assess whether the answer uses the supplied context correctly and stays grounded in it.

This separation localizes errors clearly. If the relevant source is missing, chunking, metadata, embeddings, filters, or ranking are likely responsible. If the source is present but the answer is wrong or incomplete, the more likely focus is the prompt, model behaviour, context preparation, or an answer gate. Without this view, the diagnosis remains vague: the system gives poor answers.

Security, privacy, and robustness belong in the test strategy

A domain-correct answer is worthless if it exposes confidential information or can be controlled through a manipulated input. I therefore extend evaluation with tests for prompt injection, permissions and tenant isolation, privacy, bias, and resilient error handling. This includes adversarial inputs: what happens with conflicting instructions, manipulated documents, or attempts to reveal system instructions?

Economic robustness matters as well. Token consumption, model calls, retrieval latency, and error rates can directly determine whether a system is usable. Cost and load testing therefore do not belong at the end of a project. They help define realistic limits, caching strategies, and stop rules early.

Non-determinism needs metrics and thresholds

LLM outputs are not fully deterministic. A single successful run is therefore not a reliable signal. For critical cases, I run tests repeatedly, look at distributions rather than isolated results, and define quality metrics with clear thresholds. A release may require a minimum rate of correct answers, zero critical data leaks, and a maximum latency, for example.

Thresholds should follow risk and product value, not wishful thinking. An internal ideation tool can tolerate different uncertainty than a system that informs customers, pre-screens applications, or prepares decisions in a regulated setting. What matters is deciding before implementation when an outcome is good enough—and when the system must deliberately abstain.

Changes need automated regression evaluation

Models, prompts, and knowledge sources change frequently. Each change can shift quality: a new model may write more elegantly but cite less reliably; an updated document may improve one answer while pushing an important passage down the ranking. I therefore protect relevant changes through automated regression evaluations against the same golden dataset.

Results should be versioned and comparable: which model version, prompt, retrieval configuration, and data snapshot were used for the run? This turns gut feeling into a traceable quality decision. Improvements become visible, and regressions can stop a rollout before they affect users.

Quality does not end at release

A test set never represents all of reality. In production, I therefore add monitoring for latency, cost, errors, refusals, security events, and quality indicators. Privacy remains important: telemetry should collect only the information genuinely needed for control and should not log sensitive content unnecessarily.

For critical decisions, human-in-the-loop also belongs in the design. People need clear information about what the system suggested, which evidence supports it, and when review or escalation is required. Good AI quality does not mean hiding every uncertainty. It means making limits visible and operating reliably within them.