As AI moves deeper into healthcare, its role is shifting from an experimental technology to a potential part of everyday clinical decision-making. But with that shift comes a critical question: how can healthcare AI earn trust without replacing human judgment? In this conversation, Rajeev Ranjan, Editor, Digital Terminal, speaks with Saurav Kasera, Co-founder, CLIRNET, on explainability, human oversight, responsible deployment and the future of AI-assisted healthcare in India.
Rajeev: As AI becomes increasingly integrated into clinical workflows, why do you believe explainability and transparency will be critical to building doctors' trust in AI systems?
Saurav: Because accuracy alone doesn't earn trust — not from doctors, anyway. A doctor needs to understand how an AI system got to a recommendation, what it actually looked at, and where it might be shaky.
Take a system that flags a chest X-ray as suggestive of pneumonia. The clinician needs to know what in that image drove the assessment, and how confident the system really is. And if the patient's symptoms don't line up with what the scan supposedly shows, the doctor needs to be able to push back on it — not just accept the output because the software said so.
Explainability doesn't mean dumping every technical detail of the model onto the clinician. It means giving them something they can actually use: the clinical factors that mattered, the supporting evidence, a confidence level, and an honest note on where the system tends to struggle.
Transparency is the wider version of that — being upfront about where the system was validated, which patient groups were actually in the training data, and where its performance drops off. Doctors should be able to interrogate a recommendation, weigh it against their own read of the patient, and decide for themselves. That's a far more durable basis for trust than "it tested well, so trust it."
Rajeev: What are the risks of allowing AI-generated recommendations to be treated as unquestionable decisions in healthcare, particularly in high-stakes clinical situations?
Saurav: The real risk is responsibility quietly shifting from human judgment to an output nobody's scrutinising anymore.
AI systems learn from whatever data they were given, and that data has gaps — biases, blind spots, patterns that simply don't hold for every patient. A model can also sound perfectly confident while working off incomplete information, and there's often no visible sign that anything's missing.
Say a system flags a patient as high-risk for a cardiovascular event based on their data. Useful information — but it shouldn't become a treatment decision on its own, without someone checking it against current medications, contraindications, prior workups, the rest of the patient's history.
There's also automation bias to worry about — the tendency to give a machine-generated recommendation more weight than it's earned, simply because it looks objective or technically sophisticated. This is genuinely well-studied at this point, and it gets more dangerous, not less, in high-stakes settings like emergency medicine, oncology, or critical care, where a recommendation that looks reasonable on screen can still be wrong for the one patient it's about, if something important wasn't captured.
AI should sit underneath the decision, supporting it — not above it, making it. The higher the stakes, the more human verification matters, not less.
Rajeev: How can healthcare AI be designed to challenge and support a doctor's thinking rather than simply provide an answer or recommendation?
Saurav: I'd argue the most useful clinical AI isn't necessarily the one that answers fastest. It's the one that makes the doctor think better.
Instead of "this is the likely diagnosis," a stronger system surfaces relevant findings, flags what's missing, raises alternative possibilities, points out something that might have been overlooked — and lets the clinician do the rest.
Picture a patient with a persistent cough and breathlessness. A system might reasonably suggest COPD given the history. But a genuinely useful one doesn't stop there — it also surfaces smoking exposure, past respiratory infections, current medications, and anything else that could point toward a different cause, and prompts the clinician to consider whether more workup is warranted.
That reframes AI from answer-generator to something closer to a thinking partner. The design principle underneath all of it is simple: augment the reasoning, don't replace it. Help doctors notice the patterns and inconsistencies they might otherwise miss, and leave the actual call where it belongs — with them.
Rajeev: What should an effective human-in-the-loop framework look like to ensure doctors remain in control of clinical decisions while benefiting from AI capabilities?
Saurav: It has to be built into the workflow itself — not bolted on as a checkbox at the end.
Take medication prescribing. If a system flags a potential drug interaction, the clinician needs to see exactly which medicines triggered it and why it might matter for this patient. From there, they weigh the actual clinical picture and accept, adjust, or reject the recommendation — not just click through it.
Clinicians should also always know when AI played a role in a decision and what that role actually was. The evidence, the uncertainty, the limitations — all of that needs to be presented in a form someone can review quickly, not buried.
Accountability has to be explicit too. AI can assist with analysis and generate recommendations, but responsibility for the clinical decision stays with the clinician and the institution — full stop.
And oversight can't stop once the system goes live. Healthcare organisations need to keep watching how it performs in practice — including where clinicians keep disagreeing with it, or where its performance quietly drifts across different patient groups. Human oversight has to be an active, ongoing part of the design, not just a doctor's name attached somewhere in the process.
Rajeev: In the Indian healthcare ecosystem, where clinical workflows, infrastructure, and access to specialists vary significantly, what challenges need to be addressed before responsible AI can be deployed at scale?
Saurav: India is a genuinely exciting opportunity for healthcare AI, and also a genuinely hard place to deploy it responsibly — both at once.
Care looks completely different depending on whether you're in a metro hospital or a smaller town, public or private, primary or tertiary. Digital infrastructure and specialist access swing just as widely. It's worth saying plainly: a large share of specialist positions at rural health centres in India sit vacant, so the gap this technology is being asked to help close is a real and significant one, not a hypothetical.
Picture a general physician in a smaller city who needs specialist input but doesn't have easy local access to that specialty. An AI tool could help that physician spot red flags early and get the relevant clinical picture organised before a referral goes out. But it still has to work inside the actual local network — the connectivity, the diagnostics on hand, the specialists who are or aren't reachable — not the idealised version available in a big-city hospital.
Interoperability is its own challenge on top of that. These systems need clean, structured clinical data to work with, and they need to fit into existing workflows rather than add another layer of documentation on top of an already stretched day.
Population diversity raises the bar further — a model trained on one demographic or setting can't be assumed to generalise, and needs real validation before it's trusted elsewhere in the country.
And data governance sits underneath all of it — privacy, security, consent, who gets access to what. This isn't a settled area yet in India, which makes it a genuine constraint on how fast anyone should be scaling.
Ultimately, scale here depends less on building a more sophisticated model and more on building something that actually holds up across how different India's healthcare settings really are.
Rajeev: As healthcare AI moves from experimentation to real-world adoption, what principles should developers and healthcare organizations follow to ensure AI remains transparent, accountable, and clinically useful?
Saurav: Start with a real problem, not the technology. Too much gets built because the capability exists, not because someone asked for it.
Cutting the time clinicians spend summarising records, for instance, is a genuinely useful application — but only if the system reliably captures what matters and the clinician can verify it quickly. The goal is a better workflow, not proof that a model can produce a summary.
From there, developers and health systems need clear ownership at every stage — data collection, model development, deployment, ongoing monitoring, and what happens when something goes wrong. Systems should be validated for the populations and settings they'll actually be used in, and their limitations documented honestly rather than smoothed over by a strong headline accuracy number.
Transparency needs to reach the clinicians using the tool too — they should know when AI is involved, what it's actually doing, and how much weight to give its output.
Monitoring can't stop at launch. A model that performed well in development can behave differently once it meets a different population, a different workflow, messier real-world data.
And the north star through all of it should stay the same: AI is there to give clinicians better information and more time with patients — not to take the decision-making out of their hands. Responsible healthcare AI won't be judged by how much autonomy it's handed. It'll be judged by how well it combines what the machine is good at with what a human's judgment is still, and probably always will be, better at.
𝐒𝐭𝐚𝐲 𝐢𝐧𝐟𝐨𝐫𝐦𝐞𝐝 𝐰𝐢𝐭𝐡 𝐨𝐮𝐫 𝐥𝐚𝐭𝐞𝐬𝐭 𝐮𝐩𝐝𝐚𝐭𝐞𝐬 𝐛𝐲 𝐣𝐨𝐢𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 WhatsApp Channel now! 👈📲
𝑭𝒐𝒍𝒍𝒐𝒘 𝑶𝒖𝒓 𝑺𝒐𝒄𝒊𝒂𝒍 𝑴𝒆𝒅𝒊𝒂 𝑷𝒂𝒈𝒆𝐬 👉 Facebook, LinkedIn, Twitter, Instagram