The Scariest Attack in Modern Software Just Happened and a Bug Saved Us All

What Actually Happened: The Attack Chain No One Saw Coming

This was not a simple hack. It was a cascading compromise that started weeks before anyone noticed.

On March 19, a hacker group called TeamPCP quietly poisoned the popular open-source security tool Trivy. Not the paid version. The free open-source GitHub Action that thousands of CI/CD pipelines use to scan for vulnerabilities. LiteLLM's build pipeline pulled Trivy without locking it to a specific version. That one configuration decision handed the attackers everything they needed.

The compromised Trivy action ran inside LiteLLM's GitHub Actions environment and extracted the PyPI publishing token stored there. With that token, the attackers needed no code review approval, no pull request and no maintainer sign-off. They uploaded the poisoned versions directly to PyPI. The security scanner meant to protect the project became the weapon that destroyed it.

On March 24 at 10:39 UTC, version 1.82.7 landed. Version 1.82.8 followed at 10:52 UTC. Both contained a malicious file called litellm_init.pth.

Here is what makes .pth files so dangerous.

Python automatically executes .pth files placed in your site-packages directory every single time the Python interpreter starts. No import statement required. No user interaction. The moment you installed the package the payload was already running in the background. Developers who never directly imported litellm were not safe either. If any of their dependencies pulled it in as a transitive requirement they were fully compromised without ever knowing litellm was on their machine.

Everything the Malware Stole

The credential harvester was systematic and thorough. It did not guess or probe. It knew exactly where to look.

The malware scanned for and exfiltrated SSH private keys and configs, AWS access keys and secret tokens, GCP application default credentials and service account keys, Azure tokens, Kubernetes cluster configs and admin credentials, service account tokens across all namespaces, git credentials, every environment variable and .env file on the system, shell history, browser-stored credentials, crypto wallet files, SSL private keys and CI/CD secrets and database passwords.

It did not stop at reading files from disk. The malware used the stolen AWS and Kubernetes credentials to make live API calls and pull secrets directly out of running cloud infrastructure. Then came stage two. After harvesting credentials it dropped a persistent command-and-control backdoor that required no root privileges and survived reboots.

The infected packages were live on PyPI for roughly three hours. Major AI frameworks including Microsoft GraphRAG, Google ADK, DSPy, MLflow, OpenHands and CrewAI all pulled the malicious version as an indirect dependency before it was removed.

The Bug That Accidentally Saved Thousands of Companies

Developer Callum McMahon was using the MCP plugin inside Cursor when something went wrong. He had not intentionally installed litellm. It came in as a transitive dependency he did not even know existed in his stack. The malicious .pth file contained a fork bomb. Every time Python started it spawned a child process. That child process triggered the same .pth file which spawned another child process. Exponential growth. Memory consumed in seconds. Machine crashed.

That crash was the alarm.

McMahon opened a GitHub issue. The LiteLLM team scrambled to rotate all credentials for GitHub, Docker, CircleCI and PyPI. The malicious packages were quarantined. The attackers responded by flooding the GitHub issue with bots in an attempt to bury the severity of what had happened and mislead developers into thinking the problem was already resolved.

Karpathy's observation is what should keep every developer up at night. If the attacker had been a better programmer, if they had written a fork bomb that ran quietly instead of explosively, the malware would have sat on machines across the entire AI development ecosystem for weeks. Silent. Harvesting. Spreading. The credentials stolen from one environment would have funded the next compromise. Which would fund the one after that.

Wiz's head of threat exposure described what this loop looks like at scale: the open-source supply chain collapsing in on itself where each compromised environment yields credentials that unlock the next target.

Why This Is Different From Every Other Security Warning You Have Ignored

Most supply chain attack stories feel abstract. This one should not.

LiteLLM is not some obscure niche package. It is the library that the AI development community built an entire ecosystem on top of. It serves as a unified interface for calling OpenAI, Anthropic, Google and over 100 other model providers through a single API. Over 2,000 open-source packages list it as a direct dependency. When you install almost anything in the modern AI toolchain there is a real chance litellm is somewhere in that tree, two or three levels deep and completely invisible to you.

This is the precise vulnerability Karpathy identified. Every package you install forces you to trust every single dependency in its tree. And any one of them can be poisoned. Not the package you chose deliberately. The package that the package that the package you chose depends on. You have no visibility into that tree and no practical way to audit it at install time.

The AI ecosystem makes this dramatically worse. Modern AI projects carry sprawling dependency trees. LLM orchestration frameworks, agent libraries, MCP server toolkits and IDE plugins all layer on top of each other. A single compromise at one node in that graph radiates outward into thousands of downstream projects simultaneously. This is not a hypothetical scenario. It happened on March 24.

What You Need to Do Right Now

If you work with Python and AI tooling this is not optional reading.

First, check your litellm version immediately with pip show litellm. Version 1.82.6 is the last known safe release. If you are running 1.82.7 or 1.82.8 treat all credentials on that machine as compromised. Rotate everything. SSH keys, cloud credentials, database passwords, API keys, Kubernetes tokens. All of it. Then rebuild the environment entirely. Do not patch over it.

For every project going forward, pin your dependencies to exact versions using == in production instead of >=. This will not stop you from pulling in a poisoned release if you happen to be on that version but it prevents silent upgrades from pulling in something bad later. Use lockfiles. pip-compile, poetry.lock or uv.lock work well. Run pip-audit or safety as part of your CI pipeline to scan the full dependency tree for known issues. Do not run pip install as root. Limiting privileges limits the blast radius of any compromise.

Pay attention to .pth files. They are a legitimate Python feature but they are also a near-perfect delivery mechanism for malware that runs without any user action. If you see an unfamiliar .pth file appear in your site-packages directory that is worth investigating before you do anything else.

Karpathy's longer-term suggestion is worth taking seriously. For functionality that is simple enough to implement cleanly, consider having an LLM generate the code directly rather than pulling in a third-party library. The dependency model was built on an assumption that the bricks are trustworthy. That assumption is getting harder to defend.

The Uncomfortable Truth About How We Got Here

The open-source supply chain was designed for a world where attacks happened at the application layer. Someone exploited your code. You patched it. The model made sense.

That model is broken now.

TeamPCP did not exploit LiteLLM's code. They compromised the security tool LiteLLM used to protect itself and then used the credentials that security tool had access to. The attack originated three levels above where anyone was looking. And it cascaded silently downward until a fork bomb turned one developer's machine into an accidental alarm system.

The next group that runs this playbook will learn from that mistake. They will not write sloppy code. They will not trigger memory explosions. They will sit quietly inside your dependency tree and your cloud infrastructure for as long as they need to. Karpathy said it plainly: the attacker vibe coded this attack and that is the only reason it got caught. Next time they will not make that mistake.

The question every development team needs to answer is not whether this can happen to you. It already nearly did. The question is what you are going to do differently before the next one.

The Bottom Line

A package with 97 million monthly downloads was poisoned and remained live long enough to compromise major AI frameworks and an unknown number of individual developer machines. The attack was discovered by accident. The attacker made a coding error that caused a crash that a developer noticed. That is the entire margin between this being a contained incident and being a weeks-long silent catastrophe affecting the infrastructure of hundreds of companies.

Supply chain attacks are the scariest thing in modern software because they weaponize the thing developers rely on most: trust. Trust that packages on PyPI are what they say they are. Trust that the security tools protecting your build pipeline are not already compromised. Trust that the dependency three levels deep in your stack was not poisoned last week.

That trust is the attack surface now.

Pin your dependencies. Audit your trees. And the next time your machine crashes unexpectedly, maybe check your site-packages before you dismiss it as a fluke.

If this post changed how you think about your dependency strategy, share it with the developer on your team who still runs pip install without a second thought. That developer is the one who needs to read this.