proof42.ai / Articles / Guide

How to Build a Test Question Set for AI Answer Accuracy

A fixed, tagged question set with a scoring rubric reveals empty results and freshness gaps before users encounter them.

By Zoltán Varga2026-11-055 min read

Why a Fixed Question Set Beats Impression Metrics

User feedback often hides systemic failures in retrieval systems. A user might ask one easy question and receive a perfect answer, creating a false sense of security. This anecdotal evidence fails to capture gaps in specific domains or the inability to handle complex synthesis tasks. A fixed question set removes this noise by testing the same ground truth repeatedly.

Impression metrics rely on what users happen to ask. They miss critical areas where the corpus is thin or outdated. A structured benchmark forces the system to answer questions it would otherwise avoid. This approach aligns with standard information retrieval practices used by search engines for decades to index and validate content.

The goal is to establish a baseline that tracks progress over time. Without a static set of questions, you cannot measure if a change to the vector store or the ranking algorithm improved performance. You need a consistent yardstick to compare the 'before' and 'after' states of your AI infrastructure.

This method shifts evaluation from subjective feelings to objective data points. It allows teams to identify exactly which types of queries fail. The result is a clear roadmap for remediation rather than vague promises of improvement.

Choosing Domains, Difficulty Levels, and Freshness Requirements

A robust test set covers multiple dimensions of knowledge. You should tag each question with a specific domain, such as business consulting, synthetic persona modeling, or core psychology. This tagging reveals which parts of your knowledge base are strong and which are missing entirely. A balanced set prevents the system from gaming the metrics by only mastering one narrow topic.

Difficulty levels must range from easy factual recall to complex synthesis. A dataset with only simple questions will show high success rates while hiding deep weaknesses. Conversely, a set of only hard questions may frustrate the team without showing incremental gains. A mix of easy, medium, and hard levels provides a nuanced view of system capability.

Freshness is a critical variable that standard benchmarks often ignore. You must distinguish between canonical knowledge found in textbooks and recent developments from the last two years. Systems trained on static data often fail on recent queries. Explicitly marking questions as 'canonical' or 'recent' exposes these latency gaps immediately.

The selection process requires defining expected sources for every question. You should list the specific books, papers, or reports that must appear in the retrieval results. This ensures the system retrieves the correct context, not just a plausible-sounding answer generated from its internal weights. The presence of the right source material is the first step to a correct answer.

• Tag questions by domain to isolate weak knowledge areas.
• Include easy, medium, and hard difficulty levels to test depth.
• Separate canonical knowledge from recent research to check freshness.
• List expected source documents for every test item.

Writing Expected Answers and a Scoring Rubric

Defining the expected answer is more than writing a single sentence. You must identify the key concepts that must appear in the response. These concepts act as the ground truth for your evaluation script. If the answer mentions the right ideas but misses a specific term, the system may still be useful, but the metric will flag it.

A scoring rubric should prioritize key concept matches over exact string equality. Exact matching is too brittle for natural language variations. A substring match on key terms within a specific window offers a practical balance between precision and recall. This method captures the semantic intent without requiring a perfect phrasing match.

The rubric must account for the difference between retrieval quality and answer generation. A system might retrieve the perfect document but fail to summarize it correctly. Your evaluation needs to separate these two failures. One metric checks if the right chunks were found, while another checks if the final text contains the required concepts.

Documentation of the rubric ensures consistency across different evaluators. If multiple people score the results, they need clear rules on what counts as a hit. Ambiguity in the rubric leads to inconsistent data, making trend analysis impossible. Clear definitions turn subjective judgments into reliable engineering metrics.

• Identify key concepts rather than writing a single expected answer string.
• Use substring matching on key terms to allow for phrasing variations.
• Separate retrieval metrics from generation quality in the scoring logic.
• Document scoring rules clearly to ensure evaluator consistency.

Reading the Baseline: Hit Rates and Empty Results

Initial runs often reveal shocking gaps in performance. A baseline test on a corpus of over two million chunks might show a concept hit rate of less than nine percent. This low number indicates that the retrieval system is failing to surface relevant information for the majority of queries. The data points to a fundamental issue with indexing or ranking.

Empty results are the most critical signal to watch. When a system returns zero documents for a question, it represents a complete failure of the retrieval pipeline. In a test set of seventy questions, finding twelve empty results highlights significant holes in the knowledge base. These gaps often occur in specialized domains or for recent topics.

Analyzing the distribution of hits reveals structural weaknesses. You might find that easy questions succeed while medium-difficulty ones fail. This anomaly suggests the system handles simple lookups well but struggles with synthesis or specific business contexts. The pattern tells you exactly where to focus your remediation efforts next.

Freshness gaps appear clearly when comparing canonical versus recent questions. If the system scores well on textbook concepts but poorly on recent research, the corpus lacks up-to-date materials. This distinction is vital for organizations that rely on current market data or the latest scientific findings. The metric quantifies the risk of using outdated information.

• Expect low initial hit rates to identify immediate retrieval failures.
• Prioritize fixing questions that return zero results.
• Analyze performance by difficulty level to find synthesis gaps.
• Compare canonical and recent scores to detect freshness issues.

Re-running After Every Change

Evaluation is not a one-time event but a continuous loop. Every change to the vector store, the embedding model, or the ranking algorithm requires a re-run of the full question set. This practice ensures that improvements in one area do not cause regressions in another. It turns deployment into a data-driven decision process.

Adding a reranker to the pipeline often yields dramatic improvements. Tests show that introducing a reranker can increase concept hit rates by more than five times compared to a baseline without one. The reranker refines the initial list of candidates, pushing the most relevant documents to the top. This step is often the most cost-effective optimization.

Speed and latency become constraints as the system scales. A reranker might improve accuracy but slow down response times significantly. You must balance the gain in hit rate against the cost in milliseconds. The evaluation set helps you find the sweet spot where accuracy meets operational requirements.

Iterative refinement leads to a mature, auditable system. Over time, the hit rate climbs as you fill corpus gaps and tune parameters. The question set becomes a living document that evolves with your knowledge base. It provides the evidence needed to trust the AI assistant with critical business decisions.

• Run the full test set after every configuration change.
• Measure the impact of rerankers on both accuracy and speed.
• Balance accuracy gains against latency costs.
• Treat the question set as a living document for continuous improvement.

AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.