PROBE: Release-Gate Evaluation for Regulated Enterprise RAG Conflict-Aware Retrieval, Grounding, and Safety Metrics on a Controlled Anonymized Benchmark
Author(s): Sandeep Nutakki
Publication #: 2606048
Date of Publication: 18.05.2026
Country: United States
Pages: 1-25
Published In: Volume 12 Issue 3 May-2026
DOI: https://doi.org/10.62970/IJIRCT.v12.i3.2606048
Abstract
Retrieval-augmented generation (RAG) systems are increasingly used to answer policy, clinical, financial, and operations questions, yet common retrieval benchmarks rarely measure whether a generated answer is grounded in the correct, current, and decision-relevant source. We present PROBE, a controlled evaluation harness for measuring retrieval, grounding, and safety quality in regulated enterprise RAG applications before release. The study uses a de-identified benchmark of 2,400 questions across supply chain, healthcare operations, and financial services, with 180 expert annotators producing 7,200 independent judgments. Within this controlled anonymized benchmark, the PROBE Composite Score reached Spearman rho = 0.72 with adjudicated business acceptability (95% CI [0.69, 0.75], p < 0.001), compared with rho = 0.47 for NDCG@10 and rho = 0.58 for RAGAS faithfulness. The PROBE unsafe-answer detector achieved F1 = 0.86 (95% CI [0.84, 0.88]) in the held-out split, while the best baseline evaluator reached F1 = 0.77. Annotators reached kappa = 0.83 on the conflict-aware grounding rubric, and the top three failure modes accounted for 71.0% of unacceptable answers. These results suggest that release decisions for similar regulated-domain RAG systems should jointly measure retrieval relevance, temporal validity, source conflict, and downstream decision risk rather than rely on retrieval rank alone.
Keywords: retrieval-augmented generation, RAG evaluation, enterprise search, grounding, information retrieval, benchmark design, safety evaluation, regulated domains, retrieval metrics
Download/View Count: 3
Share this Article