# Why government AI pilots stall, and the four things that get them to production

> Almost every agency has run an AI pilot. Far fewer have anything in production. The blockers are consistent, predictable, and almost never about model quality.

- Type: Article
- Published: 2026-07-22
- Author: Black Mountain AI
- Topics: Delivery, Government, Production AI
- URL: https://www.blackmountainai.com.au/insights/why-government-ai-pilots-stall

Nearly every Commonwealth agency has now run an AI pilot, and a much smaller number have a system that public servants actually use on a Tuesday afternoon. The interesting question is not why the pilots succeeded, since modern models make a convincing demonstration easy, but why so few crossed into production.

Having been called in to rescue or restart a number of these, we see the same four failure points. None of them are about whether the model was good enough.

## The pilot was built outside the environment it would have to live in

The fastest way to demonstrate value is to stand something up in a sandbox, or on a laptop, or in a vendor's cloud with a sample of exported data. This is a reasonable choice for a two-week experiment and a fatal one for anything after that.

The problem is that almost everything difficult about the production version lives in the environment: how identity and authorisation work, where data is allowed to sit, what the network will permit, how secrets are managed, which platforms your organisation has already accredited, and what your change process requires before anything is deployed. A pilot that skipped all of that has not de-risked the project. It has deferred the risk and created a false impression that the hard part is done.

The fix is uncomfortable but simple: build the pilot inside the real environment, or at minimum inside a faithful replica of it, and accept that this makes the pilot slower. You are buying information about feasibility, and feasibility is an environmental question.

## Nobody could say how often it was right

This is the most common and most damaging gap. A pilot gets demonstrated on a dozen hand-picked examples. It performs well. Everybody is impressed. Then somebody in the room asks how often it is wrong, and what happens when it is, and there is no answer, because no test set exists.

Without a measured baseline, several things become impossible at once. You cannot tell whether a change to a prompt, a model or a retrieval strategy made the system better or worse. You cannot give an assurance function anything to assess. You cannot set a threshold for when a human must review an output. And you cannot detect degradation when a model provider updates something underneath you.

Building a task-specific evaluation set is unglamorous work. It usually means sitting with subject matter experts and labelling a few hundred real examples, arguing about the ambiguous ones, and writing down what "correct" actually means for this task. That argument is itself valuable. It frequently reveals that the business rule the system is meant to apply was never written down clearly.

## The obligations were treated as a final gate rather than a design input

Privacy, security classification, records management and human oversight requirements are not paperwork. They are architectural constraints, and they are much cheaper to satisfy in a design conversation than in a remediation project.

We have seen a working system discarded because the retrieval index it depended on could not satisfy records disposal requirements. We have seen another need a substantial rebuild because personal information had been embedded into a vector store from which it could not be selectively deleted. Both would have been avoided by a thirty-minute conversation with the right person during design.

Bring your privacy officer, your records manager and your security assessor into the design phase. They are not there to say no, they are usually relieved to be asked early, and they know things about your obligations that will change your architecture for the better.

## The people who would use it were not involved in building it

An AI system that produces a technically accurate output that arrives in the wrong place, in the wrong format, at the wrong point in someone's workflow, will not be used. This sounds obvious and it is why a good number of otherwise sound systems quietly die after go-live.

The staff who do the work know where the friction actually is. They know that the summarisation step is not the bottleneck. The bottleneck is finding the three attachments referenced in the email. They know which parts of a process require judgement they would not delegate. Involving them changes what gets built as well as how it lands.

## What this adds up to

None of these four blockers are technical problems in the interesting sense. They are all a consequence of treating the pilot as a technology question rather than a systems and organisational one.

The practical implication is that the sequence matters. A short, deliberately scoped assessment that establishes the environment, the evaluation approach, the obligations and the users, before anybody writes a prompt, is not bureaucratic overhead. It is the difference between a pilot that becomes a system and a pilot that becomes a slide.