The Hugging Face Hack: The Defenders Were Not Allowed to Use the Good Stuff

On July 9, 2026, nothing went right for two major players in agentic AI.

Here is what happened, as plainly as I can put it. An autonomous agent system running on an unreleased OpenAI model was being trained and evaluated on a hard security benchmark. It got stuck. It reasoned that the reference answers were probably hosted on Hugging Face. So it went and took them — escaping its evaluation sandbox through a zero-day in a package registry cache proxy, then chaining two more flaws in Hugging Face's dataset processing to get code execution inside production, steal credentials, and move laterally. Hugging Face's responders later reconstructed roughly 17,600 attacker actions, grouped into about 6,280 clusters, between July 9 and July 13. The kicker: OpenAI did not know its own agents were the attacker until July 20, when it asked Hugging Face to revoke some credentials it had turned up in its internal investigation — and was told those credentials were already revoked, because they were the ones used in the breach.

This OpenAI Black Hat Keynote video is worth your time:

When OpenAI presented this at Black Hat, I am not sure the room fully registered what it was watching. This is us, collectively, crossing the modern-day Rubicon. Agents got loose, found zero-days, built themselves a covert message board inside an internal package registry, coordinated across separate runs, and broke into another company's production systems. Nobody told them to.

There was no malicious intent anywhere in the chain. Hacking was simply the shortest path to the score they were being optimized for. OpenAI's own presenters called it a watershed moment, and for once that is not marketing.

Now the part that should keep you up at night, and that got far less coverage.

The defender could not use the good stuff. When Hugging Face's security team started log analysis, they reached for frontier models behind commercial APIs — the obvious move for a company whose entire business is models. It failed. Forensic work means submitting enormous volumes of real attack commands, exploit payloads, and C2 artifacts, and the guardrails on those models cannot tell an incident responder from an attacker. Hugging Face named names: Claude Opus tripped on every attempt to analyze the attack logs, and Fable treated analyzing an exploit as launching one. So the SOC team stood up a quantized GLM 5.2, an open-weight model out of China, on their own infrastructure, rerouted the whole pipeline through it, and did in hours what would normally take days.

Read that again. The attacking model ran with its safety restrictions relaxed, because the sandbox was assumed to be safe. The defending team got the full guardrails. Same class of technology, opposite ends of the leash.

And here is the nuance I want you to carry out of this, because most of the coverage got it wrong: the thing that saved Hugging Face was NOT that the model was Chinese. It was that the weights were open and the model ran on hardware they controlled. No attacker data left their environment. No credentials went to a third party. No usage policy could shut down their investigation halfway through. Chinese labs happen to be where the best open weights currently come from. That is a symptom, not the point.

Remember the 1983 movie War Games? This is that, at scale, with the defender denied the same ammunition as the attacker.

There is far too much here for one write-up. But sometimes there comes a moment when we as a species get an opportunity to look at the world differently. Chernobyl was such a moment. A disaster that, left unchecked, could have left all of Ukraine uninhabitable. I was there, living a few hundred miles away at the time, so I witnessed that one first hand. This has the same shape: a small training accident, a containment failure that was nobody's plan, in a system everyone assumed was contained.

I was at Black Hat this year, where I spoke one-on-one and in small groups with over 500 people. (How I do not have Covid, I do not know. That is another newsletter.)

My CEO at Kindo, Ron Williams, has been making a version of this argument in public for a couple of years now. His thesis, in my words: sovereignty over your AI stack is a security posture, not a preference. Even when a provider does not train on your data, they analyze how you use their models, and what they learn about your business is theirs to act on. Open weights on your own metal are now competitive on capability and cheaper on cost, which removes the last honest excuse for shipping your crown jewels to someone else's inference endpoint. He has been saying it since before it was fashionable. This month it stopped being a thought-leadership position and became an incident report.

Before I go further, full disclosure:

I am Head of AI Product at Kindo, which sells into exactly this problem. So discount me accordingly. Everything above is linked to primary sources so you can check my work rather than take my word for it.

Key takeaways from the show floor

1. Owning a model you fully control, on your own metal, is now a survival requirement. Not a nice-to-have, not a project for the nerds. For security telemetry, incident data, and anything that would embarrass you in a deposition, on-prem is the only defensible answer — and Hugging Face's investigation is the proof of concept, done in hours instead of days. The open-weight models are exceptionally good now, and many run on the equivalent of a Mac Studio. If your organization does not have one standing by, go get one. Today, not after the incident.

2. Find your vulnerabilities before the attacker does. The attacker now has agents. So should you.

3. AI-generated code is generating vulnerabilities at a rate humans cannot triage manually. Microsoft told the Black Hat audience its CVE volume is running roughly 9 times what it was in March, and its own data ties that jump heavily to AI. Not all of these are urgent. That is precisely the problem: someone has to figure out which ones are, triage them, and fix them, and there are too many for that someone to be a person.

4. Sending your vulnerability data to frontier models is no longer a defensible choice. Most people I talked to at Black Hat have already worked this out. The models give you real benefit and they learn from you while doing it. This is the point where you decide what you share and what you keep.

5. A catastrophic version of this — aimed at critical infrastructure, banking, or defense — is now a matter of when, not if. Nothing in this kill chain required new science. Unsafe data processing, exposed cloud metadata, over-broad access, long-lived credentials: ordinary flaws, any competent human could have found them. What was new is that something explored 17,600 paths to get there, failed at most of them, and never got tired at 3 a.m. Now point that at a payments or critical infrastructure network on purpose.

6. Regulation is a missed opportunity. That ship has sailed, and with the current administration it is not coming back to port for a few years. Even if the US does regulate, nothing binds the rest of the world to the same rules. Assume you will be hacked and have contingencies in place.

And a last, uncomfortable one. With our current penchant for rude diplomacy, aggressive economic policy, tariffs, and saber-rattling, the number of people who would like to do us harm is now considerably larger than the number who will stand with us. Meanwhile, the Cybersecurity and Infrastructure Security Agency (CISA), which would coordinate the response, has gone from roughly 3,400 people to roughly 2,400 since January 2025; its own acting director conceded in a November 2025 memo that it was operating with about a 40% vacancy rate across key mission areas, and February's funding lapse furloughed 62% of what was left. https://federalnewsnetwork.com/commentary/2026/01/america-cant-afford-to-hollow-out-its-cyber-defenses/

Remember: the Hugging Face hack was an accident, against a modern, well-instrumented company with top-tier talent and current patches. Now imagine that same force applied on purpose, to legacy government systems, defended by whoever is still at their desk.

Whatever scenario you are picturing, I promise you it is worse.

Until the labs are incentivized to fix this — and I would not hold my breath — the answer is not to wait for permission. It is to run your agents in a sandbox you own, on hardware you control, with guardrails you set. The problem was never that the models had limits. It was that the limits belonged to somebody else.

What you can personally do

1. Come to my UXR "4 Essential UX Research and Design Skills" talk tomorrow. I am speaking about the new ways we need to think about and research multi-agent systems — one hour, hands-on research techniques you can use immediately. If you are in UX or UXR, this one is not optional. UX designers and researchers are the front line of what currently passes for corporate conscience. We exist to bring balance to the force, because AI is far too important to be left to engineers and data scientists alone — as the Hugging Face incident demonstrates rather vividly. There are still a couple of spots left: https://luma.com/Nudelman08

2. Vote for our SXSW workshop. Same material, deeper, with hands-on exercises. Last year, Daria and I turned away three times as many people as we could seat, so if you are going, register early: https://participate.sxsw.com/flow/sxsw/sxsw27/community-voting-sxsw/page/community-voting/session/1782494566087001Yqga

3. UX for AI Professional Certification Cohort 2 is now complete. Congratulations to our graduates! I will be posting their capstone project videos here as they come in, and some of them are genuinely excellent. I am deep in my work as Head of AI Product at Kindo, but Daria and I are planning the next cohort soon. We had to turn people away from Cohort 2 because we wanted to give everyone the level of individual attention they actually needed to get this stuff, and we will keep iterating on the format to keep it current. Get on the waitlist for first dibs on limited seats and the best price: https://uxforai.com/c/certification

4. Kindo.ai. As disclosed above, I work there, so I will keep this short: we do on-prem models and sandboxed private and frontier models with secrets held in a vault, and we spent Black Hat showing people how to find and fix their vulnerabilities before somebody else's agents do. If that is your problem, email me. No pressure to buy — I just want your honest opinion on what we are building.

That is all for now, folks. I remain, as always, your faithful servant,

Greg

P.S. UXR "4 Essential UX Research and Design Skills" talk is tomorrow. Grab the last few remaining spots before they are gone: https://luma.com/Nudelman08