Pacing Trust
On Hugging Face, Runaway AI, and what being a technology leader is really about
A key hallmark of a truly great technologist is the understanding that technology itself is not the primary focus of their work. Instead, their true focus is the organization or community they serve. What makes this perspective so vital? While technology can be deployed or adopted almost instantaneously—limited only by the speed at which data flows from one point to another—successful technology implementations depend on something else entirely. A skilled technologist recognizes that the true gauge of technological innovation is how rapidly and willingly the culture using that technology adapts to it.
Over the past several years, I have thought a lot about how to drive meaningful change within my organization to improve our workflows—especially those that rely on technology my team and I are responsible for purchasing, supporting, and enhancing. A prime example of this challenge involves a handful of products my organization uses from a single company1. Personally, I find their products outdated and incredibly difficult to support due to their lack of cloud-based infrastructure. More frustratingly, the company tends to avoid engaging with technologists, preferring to sell to practitioners instead (I suspect this is because they want to avoid criticism of their technology).
Despite my distaste for this company, I also recognize that staff who use their software, especially those with long careers in libraries, feel it meets their needs and is part of a robust ecosystem of similar librarians. So, despite my desire to burn everything this company puts out to the ground, I continue to pay the company regularly, knowing full well that another company would listen to the critical feedback, think innovatively about what an institution requires, and build products that usher organizations into the future.
I see a similar problem in the AI sector (and social media, but that’s another story for somebody else’s Substack). AI companies aren’t really identifying the problems their technology is trying to solve. Nor do they care about the problems their technology is causing. Ideally, a technologist would step back and consider what their community needs to handle the change—what is deficient within the organization, what is missing entirely, and what is a quick win to better enable the organization to weather future change. And that brings me to the OpenAI and Hugging Face Security Incident.
OpenAI and Hugging Face Security Incident
(Note: I’m about to use a lot of technology terms. I hate assuming that everyone knows them, so I’m going to do my best to define them in the footnotes.)
If you haven’t heard, during a recent OpenAI safety evaluation (with relaxed cybersecurity guardrails, I might add), two AI models—including an unreleased prototype and GPT-5.6 Sol—managed to escape their sandbox environment2. Rather than completing their assigned security challenges in the ExploitGym benchmarking suite3, the models tried to circumvent the tests by fetching answers directly from Hugging Face, a well-known repository for AI data and models. Basically, the AI decided to cheat rather than do the work (it’s really thinking like a human now!). The models obtained internet access by exploiting a zero-day vulnerability4 in a tool called Artifactory, an internal package proxy5 produced by JFrog. Exploiting this vulnerability gave the models what they needed in order to compromise Hugging Face’s production systems.
After the AI agent established an internet connection, it deployed two main injection techniques against Hugging Face’s infrastructure. First, it leveraged another zero-day vulnerability, this time in Hugging Face’s infrastructure, to harvest internal source code and access tokens6 from the environment. Next, the agent used code injection7 to run remote code directly within Hugging Face’s production Kubernetes pods8. This campaign spanned 4.5 days and comprised over 17,600 automated operations, during which the agent navigated internal networks, breached databases, and collected access tokens to inspect internal code repositories.
To ensure it could consistently issue remote commands to compromised systems without being blocked, the AI agent constructed an ad hoc communication protocol using basic public tools such as network monitoring utilities and code-sharing pastebins. It concealed its operations by using custom encoding and dynamically modifying DNS configurations9 whenever its outbound paths were restricted, a common tactic technology teams employ when an application is under attack. Ultimately, the intrusion was detected by Hugging Face’s security monitoring data and AI-driven anomaly detection tools, enabling engineers to isolate the affected systems, revoke compromised credentials, and patch the underlying vulnerabilities.
Oh, and did I mention that all of this is completely illegal? That’s right—if I were to take the actions I just described, I would be facing federal prosecution. But in this case, the AI hacked Hugging Face so it gets off scot-free.
Did something I say spark an idea?
This underscores exactly why technologists must pause and evaluate the broader societal consequences of the tools they build and deploy. If AI founders took a moment to reflect, they would recognize that we are already struggling to navigate legal frameworks around technology, economic responsibilities, and the environmental impacts of rapidly changing technology. Never mind that we don’t have some of the basic laws to resolve technology disputes. This is why technologists must take a beat to carefully weigh these critical factors before implementing any technology.
While advancing AI research may seem essential, we cannot ignore that these companies are profit-driven corporations rather than altruistic research labs, regardless of how much they try to pass themselves off as benevolent or safety-focused. True safety comes from slowing down and anticipating the long-term repercussions of our actions—a basic lesson we routinely instill in our children. Why, then, are we excusing corporate executives from these fundamental expectations of accountability?
If you have ideas, please reach out.
I’d love to hear what you are thinking or what you’d like me to write about next.
At the same time, we must demand more from our government. We cannot assume that these corporations will always act in everyone’s best interest. A government needs to be more powerful than the largest corporation it seeks to govern. This will enable it to regulate effectively and address critical issues that private entities often choose to ignore. However, what we are currently seeing is governments falling behind—ones that fail to understand technology and refuse to engage with it in ways that would enable effective regulation. And this criticism is not leveled solely at the United States; it is a global trend. Worldwide, we see governments burying their heads in the sand when it comes to technology, and even the most technologically advanced societies are failing to confront the genuine dangers and havoc that technology can unleash.
And Then the Frontier Labs Cried Out for Help
An open letter titled “Pacing the Frontier” has recently emerged, signed by approximately 1,300 employees from leading frontier AI companies.
Their primary concern is that many of these labs are on the verge of automating AI research, a breakthrough that could accelerate development far beyond its current pace and push AI beyond human comprehension or control. The signatories highlight that individual nations and corporations face intense competitive pressure (created by these labs, I might add) to avoid slowing down unilaterally, while society currently lacks the technical and governance mechanisms to intentionally pace AI’s progress. Consequently, they are calling on the U.S. government to support an international initiative to develop the tools needed to regulate automated AI development. Prominent industry figures who have signed the petition include Jakub Pachocki (OpenAI), Dario Amodei (Anthropic, formerly OpenAI), Shane Legg (Google DeepMind), Ilya Sutskever (SSI, formerly OpenAI), Shengjia Zhao (Meta AI), and John Schulman (Thinking Machines). It’s important to point out that many of these signatories are responsible for AI development in the first place.
None of the information in this letter is truly groundbreaking. Anyone observing society at the time these firms launched could have anticipated these exact dilemmas. Public trust in government began to decline in 2001, years before OpenAI opened its doors in 2015.
A primary concern in the AI industry centers on China and its rapid pace of development; in terms of AI advancement, numerous industry experts believe the distinction between American and Chinese AI firms is virtually indistinguishable (gift link). Naturally, these same analysts contend that China is merely distilling existing American models into novel iterations (gift link). Yet, the letter indicates that if US companies chose to decelerate, China would pull into the lead, though that line of reasoning is imperfect. The reality remains that a significant portion of prominent AI researchers are Chinese nationals, a fact mirrored among the individuals who endorsed the “Pacing the Frontier” letter. In short, the narrative that this is an uncontrollable external arms race is a myth created by the very people driving it, which, by the way, is a convenient distraction from their own decision to release technology into a world unprepared for it.
The signatories of “Pacing the Frontier” are finally confronting the lesson every practicing technologist learns early on: technology moves at the speed of light, but humans, governance, and culture move at the speed of trust. By treating speed as the ultimate virtue and ignoring society’s readiness, these labs created the very crisis they are now begging governments to solve for them. Real technology leadership isn’t about running as fast as possible until you fall off a cliff and cry for help. It’s having the discipline to align the pace of innovation with the capacity of the community it is supposed to serve. If the frontier labs can’t learn that basic lesson, no open letter in the world will save us from the mess they’ve built.
The name of said vendor has been intentionally left out, but if you know me in real life, you know who my least favorite vendor is in the library space because I complain about them ALL the time (hint, I haven’t actually mentioned them here yet). It’s so bad I have even gone so far as to lobby other companies to build similar tools, and I’ve developed a business plan on how to build an open source business that competes with one of their products (ok so the business plan only exists in my head, but message me if you want to learn more).
Sandbox environment: a sandbox is a safe, isolated computing space that allows people to test new code, software, hardware, and other computing technology without fear of viruses or other harmful things harming a production environment.
Benchmark suites: similar to testing suites, benchmark suites contain a collection of standardized tests, speed tests, workloads, and other measurement tools that allow users to compare the performance of hardware or software under controlled conditions.
Zero-day vulnerability: a previously unknown software or hardware flaw that the creator of said hardware or software has zero days to patch because it’s already been exploited by bad actors.
Package proxy: an intermediary server (or in this case an internal server) that sits between a developer’s system and public software registries to manage, cache, or secure code or software dependencies.
Access token: a digital key or credential that authorizes access to specific computing tools or resources on behalf of a user
Code injection: a security flaw where bad data is input into a system and makes a program run unauthorized code. Common types include SQL injection, command injection, and cross-site scripting (XSS). It happens when an app trusts user input without first checking it.
Kubernetes: an open-source platform that automates deploying, scaling, and maintaining containerized software across clusters of computers. Its basic building block is a pod, which wraps one or more containers into a single managed unit rather than running individual containers directly on the infrastructure.
DNS: Domain Name System, the system that translates web addresses into numerical IP addresses.
If the vendor is Springshare, I wholeheartedly agree. They are the worst.