Every AI pilot result is worth having
Many organisations have convinced themselves an AI pilot only counts if it delivers good news. It doesn’t, and believing otherwise is one of the most expensive mistakes a business can make.
We run pilots in every corner of our lives and we don’t apologise for them. You test drive a car before you buy it. No dish makes it onto a restaurant menu without time in the test kitchen. Hollywood screens films to preview audiences before wide release. Medicines only reach patients after controlled trials. In each case, a negative result can be just as valuable as a positive one, helping inform better decisions, and if the idea is still sound, you fix what’s broken and test again.
So why are organisations almost embarrassed when an AI pilot surfaces problems?
The logic behind companies burying difficult findings is somewhat understandable. There are budgets to justify, stakeholders to satisfy, and a technology landscape where everyone seems to be announcing breakthroughs on a daily basis; but burying these findings undermines the whole exercise. Pilots exist to inform decisions, not simply validate them.
Treasury took part in the Australian Government’s trial of Microsoft 365 Copilot in 2024. An evaluation by the Australian Centre for Evaluation found no clear evidence the tool improved work outcomes over the trial period, with staff expectations before the trial exceeding their actual experience. Treasury published the findings anyway, along with the areas where the tool showed promise and the lessons for any future rollout. This is exactly what a pilot should deliver.
Data from negative outcomes is still valuable data. In a world trying to work out how to deploy a technology expected to reshape every layer of society, it may be among the most valuable insights an organisation can discover.
A pilot telling you a particular AI tool doesn’t integrate cleanly with your existing systems has just saved you from a far costlier implementation. A pilot revealing your team isn’t ready for the workflow changes AI demands has handed you a change management roadmap. A pilot exposing bias or accuracy problems in a model before it influences real decisions has potentially saved you from something much worse than a delayed rollout.
A good AI pilot should test more than whether the model can perform the task: it should expose whether the data is ready, whether systems can be connected, whether employees will adopt the new workflow, whether the right governance is in place and whether the economics still work at scale. Too often, pilots prove capability and leave production to test readiness. The best pilots test both.
Knowing what doesn’t work is half the battle, and you can often only know for sure by testing. That said, not every ‘failed’ pilot is useful, and it is more important that organisations approach the exercise with the right methodology.
There is a big difference between a pilot with a clear purpose and one without. A good pilot starts with a specific question, such as whether a tool can cut claims processing time by 30%, and before it begins, the team agrees what success and failure look like and when they’ll make a call. If the answer turns out to be ‘no’, the organisation still walks away with valuable data. The pilots that waste time and budget never had a clear question to begin with; they run for months with no end date, get quietly shelved and leave nothing useful behind.
Some organisations will get it right the first time, which proves the idea works and gives them the case to scale. Others will uncover problems, whether that’s errors in the AI itself or gaps in their own systems and processes, which shows them what to fix while the stakes are still low.
|
You don’t have to be the fastest or commit at scale straight away. Pilots are about genuine experimentation, designed with clear questions, honest evaluations and a willingness to learn from every outcome. A pilot confirming a hypothesis is useful. A pilot challenging it might be worth more. The only truly wasted pilot is the one where the findings get ignored because they weren’t the answer anyone wanted to hear. |
![]() |
From technology owner to orchestrator: how AI is reshaping the CIO role
Stewardship and orchestration are now becoming the core CIO leadership disciplines.
Managing the increasing risk of artificial unintelligence
With AI, long-term success will depend on strong foundations, getting the fundamentals right, and...
Democratising governance: supporting every employee with responsible, secure AI
When employees have tools that help them recognise and communicate risk, governance becomes less...

