Three risks to consider before using AI agents in education

Imagine a digital assistant that can look up relevant information, compare options, draft a response and pass it to a colleague for approval. That is the promise of AI agents, sometimes called agentic AI.

Portrait of Richard Ogundele
Richard Ogundele

Unlike a standard chatbot, an agent can work through a task in several steps. Depending on how it has been designed, it may use connected tools, apply rules, send messages or trigger part of a workflow.

That makes agents exciting for education. They could support students, streamline administrative work and help researchers handle complex tasks. It also means we need to be clear about what they can do, what information they can access and who is responsible for the outcome.

At AI@Kent, we help people explore new AI tools and think through how they can be used responsibly, in line with the University’s AI principles.

In one of our recent AI Agents and Workflows: Opportunities and Challenges for Education webinars, we shared three governance risks that institutions need to address when piloting AI agents. Richard Ogundele, our Lead AI Platform Engineer, led the session.

Watch the webinar on YouTube.

  1. Accountability gaps

An agent gives a student incorrect information about a deadline. It sends a member of staff the wrong policy guidance. It makes a recommendation that affects someone’s next step.

Who owns the consequence?

We looked at the Air Canada chatbot case, where a customer received incorrect information about a refund policy. The airline was held responsible for what its chatbot had said.

The lesson is straightforward: every AI agent needs a named accountable owner before it launches. That person should understand its purpose, approve its use and be able to act when something needs to change.

  1. Data boundary breaches

AI agents can only work with the information they are given. The difficulty is that, once systems are connected, data can move into places it was never meant to reach.

A clear data boundary is essential. For each agent, we recommend a one-page statement setting out:

  • what information is in scope;
  • what is explicitly excluded;
  • who can query the agent; and
  • what data is retained.

This is not paperwork for its own sake. It gives everyone involved a shared understanding of the agent’s limits before those limits are tested in practice.

  1. Autonomy creep

An agent can begin by helping someone make a decision, then gradually take on more responsibility than anyone intended.

We recommend agreeing the right level of human oversight from the start:

  • Human in the loop: a person approves the agent’s output before action is taken.
  • Human on the loop: a person oversees the agent and can intervene.
  • Human in command: a person retains authority over when and how the agent is used.

The appropriate level will depend on the task, the potential impact and the people affected. It should also be reviewed regularly, because an agent’s role can change as a pilot develops.

A practical way to start

Human in the loop principle

During the webinar, we introduced D-COST, a simple framework for reviewing every AI agent:

  • Data: What can it see?
  • Compliance: What rules does it need to follow?
  • Operations: How will it run in practice?
  • Strategy: Who owns it and why does it exist?
  • Technical: How is it built and maintained?

The framework gives teams a practical starting point for conversations that can otherwise feel abstract. It also keeps governance connected to the work of designing, testing and improving an agent, rather than treating it as a final approval stage.

AI agents may help universities do more, but they also make it more important to decide where responsibility sits. As Richard put it, ‘Innovation and governance are not opposing forces. They’re a sequence.’ Good governance gives a promising pilot the clarity and confidence it needs to grow.

This article was generated from a presentation transcript with the help of AI, and checked and approved by the AI team. Humans were fully in command of publishing it.