# Evaluating AI systems in the public sector > If you cannot say how often a system is right, you cannot deploy it responsibly or maintain it. This is a practical guide to building the evaluation suite that makes public sector AI defensible. - Type: White paper - Published: 2026-08-05 - Author: Black Mountain AI - Topics: Evaluation, Governance, Production AI - URL: https://www.blackmountainai.com.au/insights/evaluating-ai-systems-in-the-public-sector > **The short version** > > Evaluation does more for public sector AI than any other single practice, and it is also the one most often skipped. This paper covers what to measure, how to build a test set with subject matter experts, how to set human review thresholds, and how to keep the whole thing running after go-live. One question ends a good number of otherwise successful AI demonstrations: *how often is it wrong?* If the answer is a shrug, or an anecdote, or a confident assertion with no evidence behind it, then the system cannot responsibly be put in front of the public or in support of a decision that affects someone. It also cannot be maintained, because nobody will be able to tell whether tomorrow's change to a prompt, a model version or a data source made it better or worse. Evaluation is what converts an impressive demonstration into an engineering artefact. This paper describes how to do it in a public sector context, where the consequences of error are borne by people who did not choose to be part of an experiment. ## Start with what "correct" means The first evaluation task is not technical. It is to write down, precisely enough to be applied by two different people with the same result, what a correct output looks like for this task. This is harder than it sounds and it is where the real value appears. Sitting with subject matter experts and labelling real examples reliably surfaces one of the following: - the business rule the system is meant to apply has never been written down; - two experienced officers apply it differently, and both believe they are right; - the "correct" answer depends on context the system has not been given access to; - the task as specified is not the task that is actually performed. Every one of these is a finding worth more than the system. Discovering that your own process is ambiguous, before you automate part of it, is a considerable service. ## Build a test set from real material An evaluation set is a collection of representative inputs with known-good outputs. For most government use cases a few hundred examples is enough to be informative, and going from zero to two hundred delivers nearly all of the benefit. Some practical guidance: **Sample from reality, including the awkward tail.** If you sample only clean, recent, well-formed documents you will measure performance on a population that does not exist. Deliberately include the badly scanned attachment, the item that arrived in the wrong format, the edge case everybody argues about, and the record from a decade ago that uses superseded terminology. **Hold some back.** Keep a portion of the set out of the development loop entirely, so you have an honest read at acceptance rather than a score that has been iterated against. **Version it.** The test set is an asset with a longer life than any model you will use. Store it in version control with the reasoning behind the difficult labels recorded alongside. **Do not put sensitive material in it casually.** The evaluation set will be copied, discussed and re-run. Apply the same classification handling you would to the production data, and prefer de-identified or synthetic variants where the task allows. ## Measure the things that matter to your obligations General benchmark scores tell you nothing useful about your task. What to measure depends on the shape of the system. For **retrieval and question answering**, the questions are whether the retrieved sources actually contain the answer, whether the response is supported by those sources, whether it fabricates anything, and above all whether it declines to answer when the corpus does not contain the answer. That last behaviour is more important in government than almost anywhere else, and it is the one most likely to be missing. For **classification and triage**, the interesting number is not overall accuracy. It is the error profile: which specific misroutings occur, what each one costs, and whether the system's confidence signal is trustworthy enough to set an automation threshold against. For **extraction**, measure per-field accuracy rather than per-document, and separate the fields where an error is recoverable from the fields where it is not. For **drafting**, accuracy is the wrong frame. Measure whether the draft contains claims unsupported by the source material, whether it omits anything material, and how much editing effort the human actually expends, which requires observing real users rather than scoring text. ## Use evaluation to set the human review threshold The most useful thing an evaluation suite produces is a defensible answer to the question: *at what confidence level does a human have to look at this?* Set that threshold from measured data and from the cost of each error class, then design the workflow so that the reviewing officer has what they need in order to disagree with the system: the source material, the reasoning, the confidence, and enough time. Human oversight that consists of an officer approving a queue of two hundred recommendations in an hour is not oversight, and it will not be treated as such by anyone reviewing the process afterwards. > Human oversight is a design requirement that you have to build for. If the reviewer lacks the information, the time or the authority to say no, then the loop is decorative. ## Keep evaluating after go-live An AI system's performance is not a fixed property. Model providers update their models. Your document mix changes. New categories of work appear. Staff start using the system in ways nobody designed for. So the suite has to keep running, which means the following. - **Regression on every change.** Any change to a prompt, model, retrieval configuration or data pipeline re-runs the suite before deployment. This is ordinary continuous integration applied to a component that happens to be probabilistic. - **Sampled review in production.** A small, regular sample of live outputs reviewed by a qualified human, with the results recorded. This catches drift that a static test set cannot. - **Monitor the operational signals.** Rates of human override, escalation, user correction and abandonment often move before accuracy metrics do. - **Re-baseline on model change.** Treat a model version change as a change requiring assessment, because it is one. ## What this gives you Beyond the engineering benefit, a working evaluation suite is what lets you answer the questions that arrive from outside the project. An assurance function asking for evidence the system is fit for purpose. An internal auditor asking how oversight is exercised. A minister's office asking whether AI was involved in a decision and how it was checked. A member of the public asking why an outcome was reached. Each of those is straightforward if you have measured your system, and very uncomfortable if you have not. ## Where to start if you have nothing If you already have a system in production with no evaluation, the recovery path is: 1. Collect a hundred real inputs from the last month of live traffic. 2. Have a subject matter expert produce the correct output for each, recording disagreements. 3. Run your current system against them and record the results honestly. 4. Categorise the failures by consequence rather than by frequency. 5. Set a review threshold from what you find, and put a regression run in front of your deployment process. A week of that work will tell you more about your system than the previous six months of it running.