Authored by Dr. Monica Trujillo, Chief Health Officer and Risk Officer at Telstra Health
Artificial intelligence is no longer a future consideration for healthcare. Across Australia, AI is already moving from pilots into clinical practice – from ambient AI scribes that reduce documentation burden to tools that support clinical decision-making and surface population health risks.
Yet many organisations are still navigating how to deploy these technologies safely at scale. Recent Australian research found that while 60% of healthcare organisations are piloting AI technologies, only 12% have deployed them across multiple care or administrative functions.
Imagine this scenario: at a rural hospital, an AI algorithm flags sepsis five hours before clinical presentation. The algorithm fires, the nurse acts, the patient improves. Success. But then the algorithm fires again, incorrectly, and the same nurse follows its recommendation. The patient deteriorates. In that moment, who is accountable? The vendor who built it? The nurse who acted on it? The organisation that deployed it? If you cannot answer that question clearly, your governance needs a refresh.
AI is a Medical Leadership Challenge
There is a misconception that AI is primarily a technology challenge. In my experience, it is fundamentally a clinical leadership challenge.
As AI becomes embedded in clinical practice, the questions leaders need to answer are no longer just about capability or implementation. They are about governance, leadership and accountability, needed to use AI safely, ethically and responsibly.
At a recent Royal Australasian College of Medical Administrators (RACMA) event on AI and Digital Leadership, I spoke about three critical distances where accountability can disappear.
Distance 1 – Data to Decision: Who owns the output? Between an algorithm’s recommendation and the care a person receives sits a series of human judgements and professional responsibility.
If that pathway isn’t deliberately designed and governed, responsibility becomes unclear. This is also where data quality becomes a clinical governance issue, not an IT issue. When data is incomplete or unrepresentative, algorithms reproduce existing biases at scale. Research showed a widely-used algorithm systematically underestimated health needs for African American patients because training data reflected existing inequity in the healthcare system. The algorithm faithfully reproduced that bias. That is why data integrity is clinical safety.
Distance 2 – Vendor to Governance: Who owns the system? Many organisations assume the AI tool will operate as the vendor designed it. But clinical use evolves. Scope expands. Workflows change. The vendor designed for one use case. Your organisation is now using it for three. If that drift isn’t monitored, an AI scribe recommending tests has quietly become a medical device requiring regulatory approval.
Distance 3 – Tool to Judgment: Who owns the gap? The more a clinician trusts a tool, the less likely they are to interrogate its output critically. That is not clinician failure – it is a governance failure. Building the conditions for good judgment must become a leadership priority. Organisations should capture clinician overrides of AI recommendations as valuable data, invest in clinicians’ understanding of failure modes, and foster a culture where questioning AI is expected, not discouraged.
The principles of clinical governance have not changed. We still need good data, capable people, clear accountability, defined escalation pathways and ongoing oversight.
What has changed is the speed, scale and complexity with which decisions can now be influenced. AI amplifies the consequences when governance fundamentals are not strong enough.
Building Trust Through Governance
Trust in AI is not created by algorithms alone. It is earned through strong governance, transparency and accountability.
AI learns from the systems and data we provide. When data is incomplete or unrepresentative, algorithms can reproduce existing biases at scale which means that data quality is not simply an IT issue, it is a clinical governance and, ultimately, a clinical safety issue.
Healthcare has spent decades developing pharmacovigilance to monitor medicines after they enter clinical practice. We will need the same discipline for AI. More than 10,000 AI-related incidents have been recorded globally since 2014.Governance cannot stop at implementation. It must be continuous.
Most technology organisations treat governance as a checkpoint rather than a discipline. We’ve spent years embedding clinical governance across the full product development lifecycle at Telstra Health, and we’re now extending that same discipline to AI, because while the technology is different, the obligation to keep it safe and fit for purpose isn’t.
The Opportunity is Significant. So is the Responsibility
AI has the potential to improve outcomes, reduce inequity, support overstretched clinicians and help health and care systems meet growing demand. But technology alone will never deliver these benefits. Success depends on the governance, culture and accountability that surround it.
On AI Appreciation Day, we should celebrate the progress artificial intelligence is making in health and care settings. But we should also recognise the leadership work happening behind the scenes – by clinicians, governance professionals, digital health leaders and health organisations – to ensure these technologies are safe and implemented responsibly.
Ultimately, the future of AI in healthcare will be shaped by the quality of the leadership, governance and clinical judgement that surround it, not by the technology itself.
The technology is ready. Our responsibility is to ensure our governance is, too.
If you lead a health organisation, ask yourself today: Do we have a clinical governance committee specifically for AI? Do we know which tools are deployed and for what purpose? Have we assessed TGA classification? If the answer to any is “no,” that is not a technology gap – it is a leadership gap. That is where your work begins.
Discover more about Telstra Health: https://www.telstrahealth.com/