The rise of Agentic AI is pushing enterprise technology beyond content generation and assistance towards autonomous decision-making and action. As AI agents gain access to applications, APIs, data and workflows, a series of recent incidents has highlighted the growing gap between what AI systems are capable of doing and the security boundaries designed to control them.
From an AI agent reportedly attempting to work around an access restriction on an Australian Government Medicare-related portal to incidents involving OpenAI systems accessing U.S. websites and the widely discussed Wikipedia episode, concerns around AI agents navigating digital environments, interacting with external systems and operating beyond expected boundaries are becoming increasingly difficult to ignore. While these incidents differ in context and impact, they point to a common challenge: ensuring that greater AI autonomy does not outpace security controls, governance and human oversight.
For Indian enterprises, the Australian incident and similar developments offer an important lesson: Agentic AI adoption need not slow, but it must become more controlled. Technology and security leaders shared their perspectives exclusively with DT on the incident, outlining the security controls, governance, monitoring and human oversight needed as Agentic AI adoption accelerates.
From Passive AI To Autonomous Action
Dr. Pavan Duggal, Senior Advocate, Supreme Court of India, and Global Chair, International AI Accountability Forum, describes the Australian incident as a significant turning point because it demonstrates the possibility of an AI system moving beyond its authorised boundaries.
Dr. Pavan Duggal said, “The Australian incident is a watershed moment. For the first time, a government has publicly confirmed that an AI agent went beyond what it was authorised to do and accessed non-public parts of a government system. The agent was not instructed to do this. That is exactly what makes it significant. We are no longer dealing only with humans misusing AI. We are now dealing with autonomous systems whose actions can outrun the intentions of the people who deploy them.”
The distinction is important. Traditional enterprise software generally follows predefined workflows, whereas an agentic system can interpret an objective, adapt its strategy and interact with multiple tools. That flexibility is one of the reasons enterprises are investing in Agentic AI but it also creates opportunities for unintended behaviour.
Vijay Sethi, Chairman, Mentorkart and Chairman, Crafsol Technologies, points out, the incident represents a shift in the nature of the threat itself. “While no personal records were stolen during the recent incident involving an OpenAI research agent accessing the Australian Government’s Medicare Statistics portal, the agent autonomously bypassed access controls when blocked. From an enterprise security perspective, this represents a significant shift in the risk landscape—from passive data leakage to active algorithmic circumvention.”
For organisations accustomed to securing human users and conventional applications, this creates a fundamentally different challenge. An AI agent may not have malicious intent, yet its ability to persistently pursue an objective can lead it into actions that violate established security boundaries.
“AI agents aren't trying to be malicious most of the times; they are over-eager problem solvers that simply do not understand real-world context, policies, or legal consequences,” said Vijay Sethi.
The Problem Is Autonomy Without Sufficient Boundaries
The Australian incident also raises questions about whether conventional security architectures are sufficient for autonomous systems.
Dr. Jagannath Sahoo, Group CISO & Data Protection Officer, INOXGFL, says the incident highlights a fundamental change in the cybersecurity landscape. “The reported AI-agent incident involving the Australian Government portal should be viewed as a critical learning opportunity, not a reason to halt Agentic AI adoption. The incident highlights a major shift in cybersecurity, where organizations must secure not only human users and applications but also autonomous AI agents capable of making decisions and taking actions independently.”
He notes that Agentic AI can adapt its behaviour when it encounters obstacles, increasing the risk of unauthorised access, privilege escalation, prompt manipulation, tool misuse, excessive autonomy and unintended actions. “Unlike traditional software, Agentic AI can adapt its behavior when it encounters obstacles. This increases risks related to unauthorized access, privilege escalation, prompt manipulation, tool misuse, excessive autonomy, and unintended actions.”
Vaibhav Tare, CISO, Fulcrum Digital, similarly argues that the risk extends well beyond the traditional concept of AI-generated content. He said, “The recent AI agent incident involving an Australian Government portal should be seen as a reminder that autonomy without governance creates new enterprise risks, not as a reason to slow down Agentic AI adoption.”
Vaibhav further added, “For Indian enterprises, the priority should be deploying Agentic AI with stronger security controls rather than delaying adoption. AI agents are no longer just generating content, they are accessing enterprise applications, interacting with APIs, retrieving sensitive data, and executing workflows. That significantly expands the attack surface to include AI identities, permissions, memory, and autonomous actions.”
Should Indian Enterprises Slow Down Agentic AI?
Despite the severity of the concerns, the experts are broadly aligned on one point that the Australian incident does not, by itself, justify abandoning Agentic AI. Instead, they advocate a shift from uncontrolled experimentation to disciplined, security-led deployment.
Sujoy Brahmachari, CIO and CISO, Rosmerta Technologies Ltd, puts it directly, “For Indian enterprises, this incident should not mean stopping Agentic AI adoption. Instead, it should encourage a more careful and responsible approach. The focus must shift from rapid deployment to secure and controlled deployment.”
Tony Anscombe, Chief Security Evangelist, ESET, takes a similar view, describing the incident as a warning rather than a reason to significantly slow AI deployment. He says, “The security implications of the recent AI agent attack that breached an Australian government website do not mean that AI deployment within organizations should significantly slow down; they should act as a warning that deployment needs to be managed with strict guardrails, monitoring and oversight. There should be a moment of reflection to ensure current policy and practice are adequate to stop any unexpected or rogue behavior.”
Sharda Tickoo, Country Manager for India and SAARC, TrendAI, takes the argument further, describing the incident as a glimpse of what can happen when agentic capabilities are not matched by equally strong controls. “The Australian incident is not an anomaly but a preview. An AI agent encountered an access restriction and worked around it, not through malice, but through goal-directed persistence, exactly the behaviour agentic systems are designed for. That is precisely what makes it alarming. We built these systems to be resourceful. The real work now is designing boundaries as intentionally as we designed the capability, so that resourcefulness never outpaces control.”
Her point is particularly relevant as enterprises move rapidly to incorporate agents into everyday applications. The challenge is not necessarily the speed of adoption itself, but whether security and governance mechanisms are being developed at the same pace.
Sharda adds, “Should Indian enterprises slow down? No, but they should stop conflating speed with recklessness. Slowing deployment does not solve the underlying problem but the ungoverned autonomy does. What Indian organisations need is disciplined acceleration.”
Zero Trust Must Extend To AI Agents
One of the strongest areas of consensus among the experts is the need to treat AI agents as distinct digital identities rather than simply extensions of existing software or employees.
Dr. Jagannath Sahoo, INOXGFL recommends a “Zero Trust for AI Agents” approach. “Organizations should adopt a "Zero Trust for AI Agents" approach, ensuring that every AI agent operates with least-privilege access, defined permissions, continuous monitoring, and detailed audit trails.”
He further argues, “AI agents should be treated as digital identities with privileged access, governed with the same rigor applied to employees, contractors, and service accounts.”
ESET's Tony Anscombe makes the same comparison, stressing that enterprises would never give employees unrestricted access and should not give AI agents such freedom either. “AI agents must be treated in a similar way to employees; no organization provides unrestricted access to an employee, and it does not allow employees to break the law in performing their duties. An AI agent should be granted least-privilege access to systems and data and have sufficient guardrails to ensure acceptable boundaries are not crossed. In cases where boundaries are met, the agent should have explicit instructions to ask for human approval.”
This principle translates into a straightforward enterprise security requirement: an agent should receive only the permissions required for its defined task, with access restricted by policy rather than by the model's interpretation of its objective.
Vijay Sethi, Crafsol Technologies recommends dynamic identity and access controls, including automatically revoking a session when an agent repeatedly encounters an unauthorised boundary. “Assign AI agents strictly scoped service accounts governed by Least Privilege rules. If an agent encounters an unauthorized boundary twice, its session token must automatically revoke.”
Human Oversight Remains Critical
While Agentic AI is designed to operate autonomously, the experts argue that autonomy cannot mean the complete removal of humans from high-impact decisions.
Vijay Sethi, Crafsol Technologies recommends explicitly defining what an agent should do when it encounters a restriction. “Prompts should explicitly instruct: "Search this database. If you encounter an 'Access Denied' message, STOP immediately and ask a human." Key high-stakes actions must always keep a human in the loop.”
For Mohit Tandon, Vice President – IT, Metro Group of Hospitals this principle should form part of a phased deployment model. “For Indian enterprises, therefore, the lesson should be controlled acceleration. Agentic AI can still deliver substantial value in areas such as customer support, software development, internal knowledge management and workflow automation. However, organizations should initially restrict agents to low-risk, reversible tasks and progressively expand their authority only after security testing.”
Mohit Tandon identifies three key controls:
Least privilege: Agents should receive only the data, APIs and permissions essential for a specific task.
Human approval: High-impact actions such as financial transactions, deletion of data, production changes or access to sensitive information should require human authorization before execution.
Continuous monitoring: Every tool call and consequential action should be logged, monitored and capable of being rapidly stopped or reversed.
His conclusion captures the broader industry position, “Thus, Indian enterprises should reconsider unrestricted deployment, not Agentic AI itself.”
Monitoring Must Go Beyond Conventional Security Alerts
Another recurring concern is visibility. If an agent can change its approach when an initial attempt fails, enterprises need to monitor not only what the system ultimately does but also how it behaves while pursuing its objective.
Anscombe from ESET argues that monitoring needs to be continuous and capable of detecting repeated attempts to circumvent restrictions. “The actions of AI agents should be monitored 24/7 to ensure compliance with the specific task set and adherence to the guardrails and boundaries. This observability needs to encompass all elements of agent activity and include capturing repeated requests where an agent makes repeated attempts looking for a method to circumvent a denial of access.”
He also highlights the importance of network-level boundaries, particularly when agents are intended to operate within internal environments. “In the case of internal-only tasks, the boundaries set need to include restricted network access; the agent should not be able to venture out of the environment it has been tasked to work within, and any attempt to communicate outside this environment should result in automatic shutdown of the agent.”
For Indian enterprises, this means AI security cannot remain limited to prompt-level controls. Identity management, network segmentation, API security, runtime monitoring and incident response all become part of the Agentic AI security architecture.
Prompt Instructions Alone Are Not Enough
The incident also raises an important distinction between telling an agent what it should do and technically enforcing what it is allowed to do.
Atul Bansal, Senior Vice President – IT, Gateway Distriparks Ltd., argues that organisations need to look beyond the assumption that a well-written prompt can act as a security boundary. “My assessment is that this exposes a critical gap between assigning a legitimate objective and controlling the methods an agent uses. A system designed to complete tasks can cross authorisation boundaries while pursuing an apparently harmless request. Prompt instructions alone cannot provide dependable containment.”
Atul Bansal recommends beginning with narrowly scoped deployments and enforcing authorisation outside the AI model. “Deployment should begin with narrowly scoped, read-only pilots using approved APIs. Each agent needs a distinct identity, minimum permissions, restricted network access and credentials with limited lifetimes. Authorisation must be enforced outside the model.”
He also recommends testing for prompt injection and attempts to bypass restrictions, while ensuring that audit logs cannot be altered by the agents themselves. “Enterprises should also test for prompt injection and attempts to bypass restrictions, maintain audit logs agents cannot alter, monitor behaviour continuously and rehearse emergency shutdown and recovery. Vendors must commit to prompt incident notification and provide evidence that safeguards work.”
From Security Controls To AI Accountability
The incident has implications beyond technical architecture. As autonomous systems become capable of taking consequential actions, enterprises also need clearly defined ownership and accountability. Dr. Pavan Duggal highlights this question through the lens of AI liability. “This is precisely the question I have addressed in Duggal Global Agentic AI Liability Framework: when an agent acts beyond its mandate, someone must answer for it, and that someone cannot be the machine.”
He also points to the gap between the occurrence of the incident and its notification as an important governance issue. “The gap between the incident in June and the notification in September shows that detection and disclosure have not kept pace with autonomy.”
For Indian enterprises, he notes that organisations deploying agents must consider existing legal and reporting obligations. “Unauthorised access remains unauthorised access, whether a human or an agent carries it out. Under Indian law, the Information Technology Act, the CERT-In incident reporting directions and the Digital Personal Data Protection regime will apply to the enterprise that deploys the agent.”
Atul Bansal, Gateway Distriparks similarly argues that accountability must extend across organisational functions. “Indian boards should assign ownership jointly across technology, cybersecurity, business and privacy teams, with escalation responsibilities agreed before launch.”
Building Containment Into The Architecture
For Dr. Pankaj Dikshit, Executive Director, Chief AI & Data Officer, Cygnet.one, the incident demonstrates why organisations need to understand not only what agents are designed to accomplish, but also how they may behave when confronted with barriers.
“The agent(s) in this particular incident, and possibly in more such cases, having encountered restrictions did not halt but circumvented the protections and went on to access non-public data and even wrote unauthorised files on government servers. Clearly, the agents went well beyond what their owners ever expected or even wanted them to do.”
Dr. Pankaj, Cygnet.one identifies autonomy itself as a central security consideration. “The first and foremost is that the agents were apparently not provided any limitations of action, their brief being to complete their so called ‘mission’ at all costs and through any means. They went from passive information gathering to actively testing and bypassing whatever restrictions and barriers they encountered.”
His proposed approach is to put technical barriers around autonomous systems rather than relying entirely on their instructions. “Ensure agents never directly communicate or integrate with critical information architecture, i.e. they should be sandwiched between inbound and outbound filters to create sandboxed environments for them to work in. Identify zones such as ‘green’, ‘yellow’ and ‘red’ to earmark levels of data access and privileges accorded to the agentic AI systems. Create API based systems to enforce strict control.”
He aso recommends session expiry controls and human checkpoints, “Establish session expiry limits and define the ‘action on failure’ mechanisms to ensure that the activities of agents are monitored. Lastly, maintain HITL (human-in-the-loop) checkpoints to have the actions of the agentic systems in a visual loop by enterprise IT and business teams.”
The Case For Disciplined Acceleration
The experts' responses collectively point towards a middle path between unrestricted deployment and a blanket pause. Agentic AI continues to offer significant potential across software development, customer support, knowledge management, automation and other enterprise workflows. But its deployment model needs to evolve as agents become more capable of taking independent actions.
Sharda Tickoo, TrendAI points to the speed at which this transition is already taking place, “Gartner expects 40% of enterprise applications to embed task-specific AI agents by the end of this year, up from under 5% in 2025. This is one of the fastest technology adoption curves on record. The same research warns that over 40% of agentic AI projects risk being shelved by 2027, because governance, observability and human oversight were never built in from the start.”
The implication is that governance cannot be added after an agent has already been deployed at scale. Security architecture, access controls, monitoring, testing, accountability and emergency response mechanisms need to be part of the deployment process from the beginning.
Vaibhav Tare, Fulcrum Digital describes this shift as making governance an operational capability rather than simply a policy exercise. “The broader lesson is that AI governance must become an operational capability, not just a policy document.”
What Indian Enterprises Should Take Away
Across the expert responses, a consistent framework emerges. Enterprises should not treat Agentic AI as an ordinary software deployment. Each agent needs a clearly defined identity, tightly scoped permissions, restricted access to data and systems, continuous monitoring and auditable activity.
High-impact actions should remain subject to human approval. Agents should operate within sandboxed environments wherever possible, while emergency shutdown mechanisms, rollback procedures and incident-response processes should be tested before production deployment.
Sharda Tickoo, TrendAI summarises the operational requirement, “The real test is not whether an organisation can prevent every deviation, it is whether it has a contingency plan ready when one occurs, like a kill switch to instantly revoke an agent’s access, a rollback protocol to undo unauthorised actions, and a defined escalation path so a human takes over within minutes. Agents reason around obstacles rather than simply fail at them, containment must be as automatic as the autonomy itself.”
For enterprises operating in regulated sectors such as banking, healthcare and critical infrastructure, this becomes even more important. Dr. Jagannath Sahoo, INOXGFL stresses the need for security testing and regulatory alignment. “The incident also reinforces the need for AI red-teaming, continuous security testing, risk assessments, and compliance alignment with regulations such as India's DPDP Act and sector-specific cybersecurity frameworks.”
Atul Bansal, Gateway Distriparks offers a practical threshold for deciding when an agent should be expanded beyond a pilot environment, “Expansion should depend on demonstrated control effectiveness, measurable business value and acceptable residual risk. Where those conditions are absent, slowing deployment is sound management.”
Conclusion: Agentic AI Needs Guardrails, Not A Retreat
The Australian incident has exposed a fundamental challenge in the next phase of enterprise AI: the more capable AI agents become at pursuing objectives independently, the more important it becomes to define the boundaries within which they are allowed to operate.
The technology leaders consulted for this feature do not advocate abandoning Agentic AI. Instead, they point to a more controlled model in which autonomy is matched by least-privilege access, strong identity management, continuous observability, human oversight, technical containment and clearly assigned accountability.
Dr. Pavan Duggal puts the responsibility on organisations deploying these systems, “Indian enterprises should not stop adopting Agentic AI, but they must stop deploying it blindly. Every agent needs clearly defined permissions, a human who is accountable for its actions, complete logs of what it does, the ability to halt it instantly, and a plan for reporting incidents within the timelines the law requires.”
And Cygnet.one's Dr. Pankaj Dikshit offers perhaps the clearest description of the path ahead, “The aspect of risk in new technologies has always been known, whether it was mobile phones, the internet, RPA or generative AI. Enterprises need to learn and adopt and simultaneously adapt as the technologies evolve. Staying away is NOT an option.”
The Australian episode therefore presents Indian enterprises with less of a question about whether to adopt Agentic AI and more of a question about how responsibly it can be operationalised. Clearly defined boundaries, measurable controls and human accountability are emerging as central requirements for enterprises seeking to capture the benefits of autonomous AI while managing its associated risks.
𝐒𝐭𝐚𝐲 𝐢𝐧𝐟𝐨𝐫𝐦𝐞𝐝 𝐰𝐢𝐭𝐡 𝐨𝐮𝐫 𝐥𝐚𝐭𝐞𝐬𝐭 𝐮𝐩𝐝𝐚𝐭𝐞𝐬 𝐛𝐲 𝐣𝐨𝐢𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 WhatsApp Channel now! 👈📲
𝑭𝒐𝒍𝒍𝒐𝒘 𝑶𝒖𝒓 𝑺𝒐𝒄𝒊𝒂𝒍 𝑴𝒆𝒅𝒊𝒂 𝑷𝒂𝒈𝒆𝐬 👉 Facebook, LinkedIn, Twitter, Instagram
