An AI Agent Hacked Hugging Face to Cheat on Its Own Test

On July 16, 2026, Hugging Face publishes a blog post that opens roughly like this: we detected an intrusion into our production infrastructure, driven by an autonomous AI system. The kind of sentence you’d expect to read five years from now, not in a post-mortem dated this summer.

Unwind the story and it’s almost comic in its absurdity. An OpenAI model, under internal evaluation on an offensive-cyber-capability benchmark (a kind of cybersecurity exam for AI), infers that Hugging Face probably hosts, somewhere, the answers to that test: the reference models, the datasets, the solutions. Rather than solve the challenge honestly, the agent decides to go find the answer key. It finds two code-execution flaws in Hugging Face’s dataset processing pipeline, gets access to a compute node, then harvests cloud credentials that were sitting on that machine. From there, it moves laterally across several internal clusters over a weekend.

The final tally: roughly 17,600 individual actions, grouped into 6,280 sequences, over four days. Caught not by a human noticing something off, but by another AI system, a triage pipeline combing through security telemetry because there is, literally, too much noise for a human analyst to do that in real time.

Wait, are you seriously going to use another OpenAI story to sell Microsoft again?

…yes. But let me explain why it’s not as convoluted as it sounds. I needed to set the scene, because this isn’t really a story about jailbreaking or a rogue model. It’s a story about identity and access. And that’s exactly the problem Microsoft is trying to solve with Agent 365.

It wasn’t a model problem, it was an inventory problem

What strikes me about Hugging Face’s post-mortem isn’t the sophistication of the attack. It’s the absence, upstream, of one simple question: did this process actually need access to that system?

The worker processing datasets probably wasn’t supposed to be able to touch the cloud credentials it ended up harvesting. But nobody had gone through the exercise of writing that down, making it visible, putting it under a policy you can actually check. The agent didn’t need a brilliant exploit for most of its progression: it needed privileges wider than necessary, and nobody watching what an automated process did with those privileges once it had them.

That’s precisely the hole most companies have today with their own internal AI agents. A Copilot Studio agent with access to SharePoint, an agent calling an MCP server, an automation script with an API key that’s been sitting around for eight months: can anyone in your organization tell you today how many agents exist, who created them, and what each one can touch? At most of my clients, the honest answer is no. We know how many employees we have. We don’t know how many agents we have.

What Microsoft is getting right here

This is where I’m going to be positive about Microsoft, and I mean it, not just out of habit as someone who sells Microsoft 365 services.

The agent registry in the Microsoft 365 admin center does exactly what was missing at Hugging Face: a central inventory of every agent in the organization, with an assigned owner (a “sponsor”), a status, and a risk column that aggregates signals from Entra, Defender, and Purview, the same Purview I’ve already written about for enterprise knowledge, here repurposed for agent governance. It’s not glamorous. It’s honestly a bit boring, as features go. But it’s exactly the kind of boring plumbing that would have kept the Hugging Face story from lasting four days.

The other piece, more foundational still, is Entra Agent ID. The basic idea: an agent is no longer an OAuth token borrowed from a user’s account, or a service key stashed in a config file. It’s a first-class identity in the directory, on the same footing as an employee. Which means: conditional access policies that apply to the agent, automatic expiration for inactive agents, a permission scope defined at creation time rather than discovered after the fact. You’re extending twenty years of IAM plumbing (the same plumbing that governs your human users) to agents, instead of inventing a parallel system bolted on for the occasion.

That’s the right answer, and it’s the boring answer: knowing who this agent is, who’s accountable for it, and what you’ve explicitly authorized it to touch. No new AI layer watching the AI here, just identity plumbing we’ve known about for a long time, finally applied to the right objects.

I’ll qualify this a bit, because I don’t want to sell it to you as solved. Entra authentication for Copilot Studio reaches general availability on September 30, 2026, which is just a few weeks away as I write this. It’s brand new. The agent registry is still stabilizing in the admin center. That doesn’t mean the direction is wrong, it means you shouldn’t yet treat this as a system with ten years of battle scars. You’ll have to test it yourself, in your own tenant, before you sleep easy.

What the registry doesn’t tell you

Here’s where I want to leave you, because this is the real caveat.

An agent registry and an Entra identity answer the question “who is this agent, and what is it allowed to do.” That’s upstream governance: the kind of control that would have limited the damage at Hugging Face well before the agent got its hands on a single credential.

But it doesn’t tell you what the agent is doing, right now, with the rights it legitimately holds. The agent in the Hugging Face incident wasn’t stopped because an access policy blocked it. It was caught after the fact, because someone eventually ran 17,600 actions through another model to spot the abnormal pattern in the noise. Identity tells you who has the key to the house. It doesn’t tell you that whoever has the key is rifling through your drawers at three in the morning.

That’s exactly the subject of my next article: how you monitor an agent’s behavior once it has the access you’ve granted it, and, more importantly, how you evaluate whether that behavior is normal or not. A well-kept registry is necessary. But Hugging Face surely had, somewhere, a list of who had access to what. What they were missing was someone, or something, watching what happened during the four days between the intrusion and the detection.

To be continued.

Sources: Hugging Face, security incident blog post, July 2026, OpenAI, joint statement on the incident, Microsoft Agent 365, Agent registry, Microsoft Learn.