Managing the increasing risk of artificial unintelligence

The current AI conversation is focused on how Australia is implementing stronger regulation to combat the dangers of increasingly powerful AI models. However, while many organisations are worried about the risks of AI becoming too capable, they are still grappling with AI that is not capable enough.

Artificial superintelligence should not be ignored, but for many organisations the more immediate issue is actually artificial unintelligence. That is, AI that hallucinates facts, overlooks critical information, misunderstands the context or recommends something that conflicts with the organisation’s policies. It’s unpredictable, uneven and often incorrect. In a low-risk setting, this might be an inconvenience, but in a business involving customer accounts, financial decisions or regulatory obligations, the consequences can be considerably more serious.

So rather than prioritising adopting more AI and more powerful AI tools, Australian organisations should first determine where AI can be trusted, what controls are needed and how to make its performance dependable at scale.

Intelligence doesn’t mean reliability

Different types of AI offer different capabilities, and understanding their limitations is essential to making sound technology decisions.

Generative AI can create content and summarise information, predictive AI uses historical data and statistical patterns to forecast outcomes and identify risks, and agentic AI can plan and execute actions to reach a certain goal. Meanwhile, the importance of workflow orchestration is coming back into focus, to coordinate tasks and decisions according to defined rules.

The more autonomous the tool is, the more important it is to keep an eye on its actions — and the issue arises when an organisation assumes that a model capable of generating a great answer is ready to make important business decisions. A system might perform well in a demonstration, but used on a larger scale and more frequently, its behaviour may become difficult to predict, especially if it is working with incomplete information, unusual requests or conflicting instructions.

This is when deploying a more capable but unreliable model into a business process creates additional risk rather than productivity. If employees must repeatedly check its work, correct mistakes or intervene when it goes off course, the efficiency gains are quickly lost. On the other hand, if errors go undetected, they can spread across connected systems and affect customers, operations and company reputation.

Reliability must be designed into the workflow

Naturally, you might think if the model isn’t working in the way you want it to, just look for a better one; but reliability also depends on the environment in which the tool operates — from the information it can access, to the rules it must follow, to the decisions it can make independently, and when it should flag something unfamiliar.

These are architectural and operational questions, and they have a huge impact on how the model performs.

For example, an AI customer service assistant may understand a customer’s question and generate a solid response, but without access to accurate account information, clear escalation rules and the right controls, the response will be generic at best, and at worst it could give incorrect advice or take an inappropriate action.

Technology leaders need to design AI around the work it must perform, rather than expecting a general-purpose model to manage every situation independently. This means providing relevant, authorised information, connecting AI to established business rules, and defining which actions require human approval. It also means recognising that some tasks are better suited to conventional automation, while others benefit from AI-assisted recommendations or human judgement.

Three considerations are particularly important, and they must all work together.

First, be deliberate about where AI belongs in your business. Figure out where it really adds value, what it can do autonomously and when a decision must be escalated, based on the consequences of a potential error.

Second, make performance measurable by testing AI against realistic business scenarios, including incomplete data and unusual requests. Monitor outputs after deployment, assessing not just accuracy but whether the system follows business rules and produces appropriate outcomes.

Third, ensure human oversight, accountability and escalation processes are built into work from the beginning. While asking employees to check every low-risk output can undermine efficiency, removing human intervention from consequential decisions can expose the organisation to avoidable harm.

What should businesses realistically expect next?

Undoubtedly, AI will continue to become more deeply embedded into everyday work and processes with growing opportunities to combine generative and predictive capabilities with existing automation.

However, progress will differ across the board. Those organisations that can clearly define beneficial uses for AI, provide the correct information, measure outcomes and manage errors will be best placed to manage this next phase of AI implementation.

Also, as regulatory expectations evolve, organisations will also need to demonstrate responsible AI use in practice, not just on paper. They must understand how systems behave, what informs their outputs, where accountability sits and how problems are identified and corrected.

For technology leaders therefore, the priority should not be to deploy AI everywhere, but to identify where it can deliver measurable improvements and establish the proper conditions for it to do so consistently. After all, long-term success — like anything — will be based on strong foundations and getting the fundamentals right, and by mitigating against artificial unintelligence to implement it safely with reliable results.

Image credit: iStock.com/patpitchaya