Analytics / AI solution providers are quickly adapting to agentic AI and pitching agentic AI solutions, ostensibly for improving productivity and increasing quality in business operations.
Figure 1: Solution providers hustling to sell agentic AI as digital leaders ponder over the implications
The Hype Cycle Is Moving Fast
Vendors and solution providers are eager to pitch agentic AI as the next big thing. They’re promising boosts in productivity, operational gains, and a competitive edge. In their zeal to be early adopters, there is a likelihood of organizations overlooking risks with agentic AI, and underestimating guardrails and change management required. In the rush to adopt, the following questions are important to consider. Are these agents aligned with institutional goals? Are the right guardrails in place? Is the organization prepared for what happens when things go wrong?
The agency problem describes the gap between principals, like owners or shareholders, and agents (people) who act on their behalf. When interests and incentives of the principals and agents don’t align, agents can make self-serving choices which may not serve to maximize the intended benefits of the people they’re meant to represent.
Now, swap human agents for autonomous AI systems with agentic AI designed to make decisions, adapt, and are meant to act independently on behalf of their principals. AI agents learn from feedback loops, evolve over time, and often operate without human intervention. Here, the risk of not always staying aligned with their goals begins. In the spirit of data science, we can express this mathematically as: –
- Principal P : The human actor (e.g., Digital or Data & Analytics Leader)
- Agent A : The agentic AI system
- Action a∈A : The action taken by AI agent
- State s∈S : The environment or system context
- Utility function for the Principal: UP (a, s)
- Utility function for the Agent : UA (a, s; θ), where θ is the agent’s internal model or learned preferences
The Agency Problem arises when:
This means the agent is optimizing for something different from what the principal actually values. We can then define misalignment as the distance (or divergence) between the objectives of the principal and the agents as:
Where D is a divergence measure (e.g., Euclidean distance, KL divergence), quantifying how differently the agent would act compared to the principal in each state.
Agentic AI systems typically use reinforcement learning (RL) or similar mechanisms where they:
- Act in the environment
- Receive feedback in the form of rewards or performance signals
- Update their internal model or policy (i.e., how they choose future actions)
The agent’s behaviour is thus not static & evolves based on its experience.
If the reward function the agent is optimizing is poorly specified, missing constraints or not representative of human (culture specific) values, then the agent may learn behaviours that are technically correct but ethically or practically harmful.
Who is really in control?
With agentic AI, the definition of “principal” has expanded. It’s no longer just the developers or the companies deploying the systems. It also includes the people whose data fuels the models, who may be structurally dis-empowered yet deeply embedded participants in AI ecosystems. These data contributors may rarely have visibility over how their information is used; yet find themselves answerable for decisions recommended by AI applications and outcomes thereof.
Agents may drift from their principal’s intent when their participation in organizational processes is not aligned in the overall organization model and their tasks are not clearly scoped. Drift and misalignment may also happen as agentic learning objectives evolve based on environmental signals and outcomes with lesser emphasis on human values. Once such systems are ‘live’ on their own, it becomes difficult to predict what they will do next making them harder to control – increasing the ‘fire-fighting’ component of the roles of the principals and their digital teams. While this problem and solution space is vast, let’s consider the following three areas:-
Understanding Consent
The way we handle consent may not be built for agentic AI. Systems may rely on older models, such as a checkbox clicked months or years ago, and never revisited. However, agents are expected to learn and update themselves all the time, which may mean that the original consent may become meaningless in systems that continue to evolve long after deployment.
We need contextual consent. People should have the ability to see how their data is used, influence how it contributes to AI behavior, and revoke access when necessary. This needs to happen not just at the start, but also continuously. Without this, the gap between data contributors and agentic decision makers only grows wider.
Build with Boundaries
As we build smart systems, we need to enable systems to recognize where their boundaries are and respect them. These boundaries need to be defined by developers from the start, with full cognizance of organizational models and business processes. Hence, roles, responsibilities, and environmental limits need to be built into the agent’s design.
Agentic AI Needs Scoped Autonomy: Not a Leap of Faith on Technology or Blind Trust on Solution Providers
We cannot expect agentic AI to act with human values unless we build those values into the system. That starts with scoped autonomy and by defining clear limits on what the agent can do and when. It also requires governance that treats contributors as stakeholders, not just data sources. Consent needs to be dynamic and reversible, behavior needs to be auditable, and decisions must be accountable.
Figure 2: A summary of challenges and topics covered in this article
Agentic AI has the potential to transform how we work and make decisions. But if we want agentic AI to serve people, rather than the other way around, we need to think deeply how these systems are designed, deployed, and governed. The agency problem does not go away when the (erstwhile human) agents become machines. It is just getting a lot more complicated.
Adopting agentic AI can become a governance challenge. If we do not ask the right questions now, we will be dealing with much bigger problems later. We should try to solve problems without creating equal or more difficult problems.
In summary:
- Scope Autonomy with Constraints: Define clear boundaries for agentic AI behavior, including role limits, escalation rules, and fallback mechanisms to prevent drift from intent.
- Align Optimization with Human Goals: Regularly audit and align the agent’s reward functions and utility models with the values and objectives of human stakeholders.
- Embed Data Agency in Design: Incorporate consent-aware policies that respect contributor preferences at both the individual and group levels, operationalizing data agency theory.
- Use Reflexive and Oversight Mechanisms: Design agents with self-monitoring capabilities and implement human-in-the-loop governance where necessary for sensitive or dynamic decisions.
- Evaluate Justice Systemically: Treat fairness, control, and alignment as system-wide justice challenges, not just model-level issues, and assess these issues using institutional and structural metrics like asymmetry and consent scope.
Disclaimer: Opinions of the author are personal & need not represent views of his employers or associates.
