
AI, Trust and Cyber Security: Research & Resources
Artificial Intelligence:
What we trust from it <> what we entrust to it
I first shared this in the form of a presentation at the SANS Security Awareness & Culture Summit 2026. A few people asked afterwards for the references behind the session, so I’ve brought together the key ideas, research and practical resources here. I'll keep updating this page as the research develops. References are linked throughout and listed at the bottom of this page.
AI is changing our relationship with technology in both directions: what we trust from it, and what we entrust to it. We're in a new decision-making environment where AI can imitate the people we trust, AI is being trusted with our information, and AI is increasingly being trusted to act on our behalf.
As AI becomes better at generating content, influencing decisions and taking action, three question areas can help us navigate that changing relationship:
TRUST
What do I believe?
When should I trust what I see, hear or read - and when should I verify?
SHARE
What do I reveal?
What information am I comfortable giving to AI, and through which tools?
DELEGATE
What do I allow?
What should AI be able to do on my behalf, and where should a human remain in control?
The answer isn't to trust nothing, share nothing and delegate nothing. It's to make better decisions when we're uncertain.
Key ideas & research
The signs we learned to look for are becoming less reliable
For years, cyber security advice has encouraged us to look for clues: poor grammar, unusual wording, an unexpected request or something that simply doesn't look or sound right. Focusing on those clues have always been in danger of lulling people into a false sense of secuirty, but now they are increasingly unreliable. We're arguably in the opposite position now, where something 'perfect' signals AI-generated and something with a few errors presents as more human.
AI can create convincing, personalised content cheaply and at scale. In a 2026 field experiment with 7,700 participants, LLM-personalised phishing achieved almost three times the click rate of generic phishing. Human-personalised spear-phishing prompted the highest percentage of clicks among recipients, but it is much more costly, in terms of labour and monetary cost. LLM-personalised phishes can be sent quickly and cheaply at scale, so while they might not get the same percentage of clicks, they get higher numbers of people clicking due to the combination of scale and persuasiveness.
The challenge isn't that we can no longer detect anything. It's that detection alone becomes a less reliable security strategy when increasingly convincing content can be generated cheaply and at scale.
A scam doesn't need to feel human, it needs to make us feel human
AI raises understandable questions about whether we will be able to distinguish a real person from an artificial one.
We focus a lot on whether people can detect synthetic content. But what if it matters less whether people can detect whether a face or voice is AI-generated, and more how the communication makes them feel? Focusing on whether people can spot deepfakes could lead us all astray.
Research into AI-powered voice phishing found that personal and emotional scenarios substantially increased willingness to comply: “The content and delivery of the scam message matter more than achieving perfect vocal fidelity”.
The challenge isn't only what AI can do to us; it's what we're doing with AI
AI is already embedded in everyday work, and use is outpacing governance:
-
Netskope reported an average of 223 detected GenAI data-policy violations per organisation per month, involving data including source code, regulated data and intellectual property.
-
ISACA reports that 83% of IT and business leaders believe employees are using AI, and yet only 31% have a formal AI policy in place
-
Seperate research from ISACA finds that only 22% of organizations provide AI training to all employees, and 32% provide no training at all
The answer to shadow AI isn't simply “stop using AI”. The lesson is broader than AI: make the safe way the easy way.
Research: Netskope Threat Labs (2025), Cloud and Threat Report: Shadow AI and Agentic AI; Netskope Threat Labs (2026), Cloud and Threat Report: 2026.
Generation was only the beginning
Generative AI creates outputs. Agentic AI can pursue an objective, take action, observe the result and adapt.
There are recent cases from July 2026 showing how this determined pursuit of the objective can have serious, unintended consequences:
-
The UK’s AI Security Institute ran a cyber security challenge 122 times and, in 10 cases, AI agents autonomously took unsanctioned actions on the live internet, including creating fake identities, social-engineering a real person and attempting to introduce malicious code. The attempts were unsuccessful, but importantly, the agents had not been instructed to deceive: deception emerged as they pursued their assigned goal.
-
During an internal cyber-capability evaluation, OpenAI agents escaped their isolated environment by exploiting previously unknown vulnerabilities, gained internet access and ultimately compromised Hugging Face’s production infrastructure. The agents chained zero-days and stolen credentials to obtain secret benchmark information, effectively discovering that cheating was another way to achieve their assigned goal.
And, of course, agentic AI adds yet more scale to the capacity of cyber criminals, to the extent that we can argue that agentic AI is a new level of persistentance in the Advanced Persistent Threat. Anthropic's report on disrupting the first reported AI-orchestrated cyber espionage campaign highlights this, with AI performing 80–90% of the campaign, including reconnaissance, vulnerability discovery, exploit development, credential harvesting and data analysis.
Design for uncertainty
We can't expect people to perfectly identify every fake, scam or unreliable AI output. And the answer isn't to (try to) stop people using AI.
Instead, we can make verification easier, put appropriate friction around high-consequence decisions, provide clear guidance for safe AI use and create cultures where people feel comfortable asking for help.
The goal isn't to remove uncertainty, it's to become better prepared for it.
Bring this session to your organisation
Jessica delivers keynotes like this for organizations around the world, exploring AI, trust, human behavior and cyber security.
Research and References
AI Security Institute (2026) Incident Report: unsanctioned agent behaviour during cyber testing
Anthropic (2025) Disrupting the first reported AI-orchestrated cyber espionage campaign
Czybik et al. (2026), A Large-Scale Study of Personalized Phishing using Large Language Models, USENIX Security Symposium.
Heiding, F. et al (2027) Evaluating AI models' capability to automate voice phishing attacks, Expert Systems with Applications, Volume 332, Part D
ISACA (2025) AI Use is Outpacing Policy and Governance, ISACA Finds
ISACA (2026) Cutting Through the AI Hype: The 2026 AI Pulse Poll
Netskope Threat Labs (2026), Cloud and Threat Report: 2026
OpenAI (2026) OpenAI and Hugging Face partner to address security incident during model evaluation