AI and safeguarding – emerging threats and practical controls 

AI safeguarding is highly topical at the moment, following recent announcements from UK government on social media (which is driven by AI algorithms), screen time, and harmful content on phones, but it’s a complicated issue, with lots of nuance. Remember, not all AI is generative AI, and AI is part of the problem, the solution, or sometimes both, as well as a topic in its own right. 

The AI Concerns 

Let’s start with the requirements. In England, the Keeping Children Safe in Education (KCSIE) guidance frames the online risks around what it calls the 4Cs: content, contact, conduct and commerce. 

Considering the 4C’s through an AI lens looks something like this: 

• Being exposed to illegal, inappropriate or harmful content: Generative AI can create harmful material on demand, including deepfake pornography and hate speech. Algorithms, on social media for example, can also push users toward increasingly extreme or radicalising content, accelerating exposure. 

• Being subjected to harmful online interactions with other users: AI chatbots and “companion” apps can pose as peers or trusted adults, building relationships that blur the line between human and machine. Offenders can also use AI to automate and scale grooming, generating believable personas using voice and video cloning and deepfakes. 

• Online behaviour that increases the likelihood of, or causes, harm: AI lowers the skill and effort needed to produce and spread explicit images of peers from ordinary photos, creating child sexual abuse material and a powerful new tool for bullying and harassment. 

• Risks such as online gambling, inappropriate advertising, phishing and/or financial scams: AI generates highly convincing phishing emails, fake websites and scam messages. AI-driven, hyper-personalised advertising and gambling algorithms can also exploit behavioural data to maximise engagement and spending. 

A layered approach to safeguarding 

When you read that all together, it can seem terrifying, but institutions already have access to tools and approaches that can support safeguarding. Understanding what is available and how to use it appropriately, in conjunction with effective human oversight to protect learners and users, is a vital first step. 

I find it helpful to view them as layers that can be stacked to create a barrier against AI harm.

The top layer consists of policy, procedure and people. Setting out clear guidance on acceptable and ethical use of digital tools that is easily available, accessible and well communicated will help avoid any confusion and support a more inclusive culture. Providing AI literacy training for staff and learners, and easy options for referral of inappropriate behaviour will also help identify and resolve issues before they can escalate. 

The digital tools used by an institution have their own inbuilt safeguards, and I’ll come to this shortly, but institutions should also consider the additional security measures they can stack around these tools. Adding extra layers such as robust user authentication, web filtering and app control alongside security information and event management (SIEM) systems and institutionally managed devices and applications can all help keep harmful AI generated content at bay. 

Within any tool that has AI enablement, it’s important to understand who is delivering that tool, how it behaves, and what features it offers as part of the package. Although the main LLM providers all have some form of moderation or safety filtering, the details vary by provider. But the broad idea is similar: the product can check the user’s prompt, the model’s response, or both, to identify content that falls into particular risk categories such as hate or harassment, sexual content or illegal activity. 

Addressing AI safeguarding at your institution 

If we are to really address AI safeguarding, we need a basic understanding of the different layers of protection available to us. Which AI tools are learners and staff actually using? Which controls sit inside the tool, which sit in the institution, and which sit with the supplier? What is blocked, what is logged, what is escalated, and who sees it? 

We then, in my view, need to fold safeguarding into broader governance and compliance planning. This perhaps won’t be easy, and I know many colleges are at very early stages in this broad area, but hopefully it helps to see it as part of the bigger puzzle. 

Colleges are well placed to take action now, but safeguarding, IT, data protection and curriculum teams need to be in the same conversation. AI safeguarding will not be solved by one product or one policy. It will be a governance issue, a technical issue and, still, a people issue. 

By Michael Webb, Director of AI at Jisc