LiteLLM Supply Chain Attack (March 2026)

A single pip install turned into a cybersecurity nightmare. On March 24, 2026, compromised versions of LiteLLM — a Python AI proxy library with 95 million monthly downloads — were published to PyPI. Once installed, a malicious .pth file executed every time Python started, silently harvesting SSH keys, API keys, cloud credentials, Kubernetes configs, and shell history. The data was encrypted and exfiltrated to attacker-controlled servers.

The irony? The malware was discovered because it had a bug that created a fork bomb, crashing a researcher's machine.

The Alert

Security researcher Callum McMahon at FutureSearch was testing a Cursor MCP plugin that pulled in litellm as a transitive dependency. Shortly after Python started, his machine became unresponsive — RAM exhaustion, CPU pinned.

He traced it to a newly installed file: litellm_init.pth — a 34,628-byte file sitting in site-packages/.

The terrifying part

The .pth mechanism fires on every Python interpreter startup. No import litellm required. Just having the package installed was enough. Every Python script, every Jupyter notebook, every Django server, every pytest run — all compromised from the moment of installation.

Impact

Library size: 3.4 million downloads per day, 95 million per month
Compromised versions: 1.82.7 and 1.82.8
Window of exposure: ~3 hours before PyPI quarantined the package
Affected systems: Any machine that ran pip install litellm during the window
Data stolen: SSH keys, .env files, AWS/GCP/Azure credentials, Kubernetes configs, database passwords, git credentials, shell history, crypto wallet files

Timeline

Time (UTC)	Event
March 19	TeamPCP compromises Trivy GitHub Action
March 24, 10:39	`litellm 1.82.7` published with embedded payload in `proxy_server.py`
March 24, 10:52	`litellm 1.82.8` published with `.pth` file — fires on every Python startup
March 24, ~11:00	Callum McMahon's machine crashes from fork bomb
March 24, ~12:00	McMahon traces crash to `litellm_init.pth`, begins analysis
March 24, ~13:00	PyPI quarantines `litellm`, all versions temporarily unavailable
March 24, ~14:00	McMahon publishes disclosure on FutureSearch blog
March 24, ~15:00	Story hits r/LocalLLaMA, r/Python, Hacker News front page

Root Cause

The Attack Chain: From Trivy to LiteLLM

This was not a direct attack on LiteLLM. It was a multi-stage supply chain attack by a threat actor tracked as TeamPCP.

The critical mistake

LiteLLM's CI/CD pipeline ran Trivy without pinning the version. When Trivy's GitHub Action was compromised on March 19, LiteLLM's CI automatically pulled the malicious version. The compromised Trivy action exfiltrated the PYPI_PUBLISH token from the GitHub Actions runner environment.

Two Different Payloads

The attacker published two versions with different delivery mechanisms:

Version	Delivery Method	Trigger
1.82.7	Base64-encoded payload in `litellm/proxy/proxy_server.py`	Fires when anything imports `litellm.proxy`
1.82.8	`litellm_init.pth` file (34,628 bytes) in `site-packages/`	Fires on every Python interpreter startup — no import needed

How `.pth` Files Work

Python's .pth mechanism was designed for adding paths to sys.path. But any line starting with import in a .pth file is executed as code on interpreter startup:

python

# Normal .pth file — adds a path
/some/additional/path

# Malicious .pth file — executes arbitrary code
import subprocess; subprocess.Popen(['python', '-c', 'exec(PAYLOAD)'])

Why .pth is so dangerous

Executes before any user code
No import of the package required
Runs on every Python process: scripts, notebooks, servers, tests
Not visible in normal import tracing
Most developers don't even know .pth files exist

The Fork Bomb Bug

The .pth payload spawned a new Python subprocess to do the actual credential harvesting. But that new subprocess also triggered .pth execution, which spawned another subprocess, which triggered .pth again — creating an unintended fork bomb.

Python starts → .pth fires → spawns Python subprocess
    └→ Python starts → .pth fires → spawns Python subprocess
        └→ Python starts → .pth fires → spawns Python subprocess
            └→ ... (exponential, machine crashes)

This bug is what crashed McMahon's machine and led to the discovery. Without it, the malware would have silently stolen credentials without anyone noticing.

The irony

The attacker's code had a bug. If the fork bomb hadn't crashed McMahon's machine, the credential stealer would have operated silently. A bug in the malware saved the community.

What Was Stolen

The credential harvester targeted:

python

# Files the malware searched for and exfiltrated
TARGETS = [
    "~/.ssh/id_rsa", "~/.ssh/id_ed25519",     # SSH private keys
    "~/.ssh/config",                             # SSH config
    "~/.env", ".env", ".env.local",             # Environment variables
    "~/.aws/credentials", "~/.aws/config",       # AWS credentials
    "~/.config/gcloud/credentials.db",           # GCP credentials
    "~/.azure/",                                 # Azure credentials
    "~/.kube/config",                            # Kubernetes config
    "~/.gitconfig", "~/.git-credentials",        # Git credentials
    "~/.bash_history", "~/.zsh_history",         # Shell history
    "~/.docker/config.json",                     # Docker Hub credentials
    "~/.npmrc",                                  # npm tokens
    "~/.pypirc",                                 # PyPI tokens
]

In Kubernetes environments, the malware additionally attempted:

Service account token theft from /var/run/secrets/kubernetes.io/serviceaccount/
Lateral movement using stolen K8s credentials
Persistence via CronJobs or modified deployments

The Fix

Immediate Response

bash

# 1. Check if you're affected
pip show litellm | grep Version
# If 1.82.7 or 1.82.8 — YOU ARE COMPROMISED

# 2. Check for the malicious .pth file
find $(python -c "import site; print(site.getsitepackages()[0])") \
  -name "litellm_init.pth" 2>/dev/null

# 3. Remove compromised version
pip uninstall litellm -y
pip cache purge

# 4. Check ALL virtual environments
find / -name "litellm_init.pth" 2>/dev/null

# 5. Install clean version
pip install litellm==1.82.6  # Last known clean version

Credential Rotation (MANDATORY if affected)

bash

# SSH keys
ssh-keygen -t ed25519 -C "rotated-after-litellm-incident"
# Update all servers, GitHub, GitLab, etc.

# AWS
aws iam create-access-key --user-name YOUR_USER
aws iam delete-access-key --access-key-id OLD_KEY_ID

# GCP
gcloud auth revoke
gcloud auth login

# Kubernetes
kubectl config delete-context COMPROMISED_CONTEXT
# Re-authenticate with your cluster

# Docker Hub
docker logout
docker login  # Generate new access token first

# npm
npm token revoke TOKEN_ID
npm token create

# PyPI
# Revoke all API tokens at pypi.org/manage/account/

DO NOT skip credential rotation

Even if the compromised versions were only installed for minutes, the malware could have exfiltrated credentials. Assume everything is compromised and rotate ALL secrets.

Lessons Learned

1. Pin Your CI/CD Dependencies

LiteLLM's CI pulled Trivy without a pinned version. This allowed the compromised Trivy to run in their pipeline.

yaml

# BAD — pulls whatever version is latest (including compromised)
- uses: aquasecurity/trivy-action@latest

# GOOD — pin to a specific commit SHA
- uses: aquasecurity/trivy-action@a7a829a0713867b681da939dc5999bbab3cee884

2. Protect CI/CD Secrets

The PYPI_PUBLISH token was accessible to the Trivy action. It shouldn't have been.

yaml

# BAD — all steps in a job share the same secrets
jobs:
  build:
    steps:
      - uses: untrusted-action@v1  # Can read PYPI_TOKEN
        env:
          PYPI_TOKEN: $&#123;&#123; secrets.PYPI_TOKEN &#125;&#125;

# GOOD — separate jobs, secrets only where needed
jobs:
  scan:
    steps:
      - uses: aquasecurity/trivy-action@SHA  # No access to PYPI_TOKEN
  publish:
    needs: scan
    steps:
      - uses: pypa/gh-action-pypi-publish@SHA
        with:
          password: $&#123;&#123; secrets.PYPI_TOKEN &#125;&#125;

3. Use Trusted Publishing for PyPI

PyPI supports OIDC-based trusted publishing — no long-lived tokens needed:

yaml

# BEST — no stored secrets, GitHub ↔ PyPI trust
jobs:
  publish:
    permissions:
      id-token: write  # OIDC token
    steps:
      - uses: pypa/gh-action-pypi-publish@release/v1
        # No password needed — uses OIDC

4. Audit `.pth` Files

bash

# Check for suspicious .pth files in your environments
python -c "
import site, os
for d in site.getsitepackages():
    for f in os.listdir(d):
        if f.endswith('.pth'):
            path = os.path.join(d, f)
            with open(path) as fh:
                content = fh.read()
            if 'import' in content and 'subprocess' in content.lower():
                print(f'SUSPICIOUS: {path}')
            elif len(content) > 1000:
                print(f'LARGE .pth FILE: {path} ({len(content)} bytes)')
"

5. Monitor for Supply Chain Attacks

Use lockfiles (pip freeze > requirements.txt with hashes)
Use pip-audit to check for known vulnerabilities
Pin exact versions in production
Use --require-hashes for pip installs
Monitor PyPI release notifications for critical dependencies

What You Can Learn

For Package Maintainers

Never use long-lived PyPI tokens — use OIDC trusted publishing
Pin all CI/CD action versions to commit SHAs, not tags
Isolate secrets — scanning tools should never access publishing credentials
Enable 2FA on PyPI accounts
Use Sigstore to sign your packages

For Package Consumers

Pin exact versions in production (litellm==1.82.6, not litellm>=1.0)
Use lockfiles with hashes (pip install --require-hashes)
Audit new package versions before upgrading
Run pip-audit regularly
Use virtual environments — limits blast radius
Monitor .pth files in site-packages

For Platform Engineers

Ephemeral CI/CD environments — don't persist secrets across runs
Separate build and publish into different jobs with different permissions
Network egress controls — block unexpected outbound connections from build environments
Package proxy — use Artifactory/Nexus to cache and scan packages before allowing internal use

The Bigger Picture

As Andrej Karpathy noted: "Supply chain attacks may be the scariest threat in modern software."

This attack succeeded because of a chain of trust:

Developers trust PyPI packages
CI/CD trusts GitHub Actions
GitHub Actions trusts third-party actions
Third-party actions trust their own dependencies

One compromise anywhere in this chain cascades to everything downstream. The LiteLLM incident is a textbook example of why zero trust must extend to your software supply chain.

XZ Utils Backdoor (2024) — 2-year social engineering campaign to backdoor Linux SSH
SolarWinds (2020) — Nation-state build pipeline compromise
Supply Chain Security — SLSA, SBOMs, Sigstore

What Would You Do?

Test your supply chain security instincts against the decisions that shaped this incident.

Scenario 1: You are a package maintainer with a CI/CD pipeline that uses several third-party GitHub Actions for scanning and publishing. You hear that a popular GitHub Action (Trivy) was compromised on March 19. Your CI/CD ran Trivy using an unpinned version. Do you (A) immediately rotate all secrets accessible to your CI pipeline, (B) check your CI logs first to see if the compromised version actually ran, or (C) assume you are safe because the compromise happened 5 days ago and you have not seen any issues?

What should have happened: (A) — immediately rotate all secrets. If your CI pulled an unpinned action and that action was compromised, you must assume every secret accessible to that CI runner has been exfiltrated. Checking logs (B) is good but insufficient — a sophisticated attacker may clean logs. Assuming safety (C) is exactly how LiteLLM's PyPI publish token was stolen and used 5 days later to publish backdoored packages to 3.4 million daily users. Pin all CI/CD actions to commit SHAs, and isolate secrets so that scanning tools never have access to publishing credentials.

Scenario 2: You just installed a Python package and your machine immediately became unresponsive — RAM exhausted, CPU pinned at 100%. You kill the runaway processes and trace the issue to a .pth file in site-packages/. Do you (A) just delete the .pth file and move on, (B) investigate the file's contents and check if credentials were stolen, or (C) wipe the machine and start fresh?

What Callum McMahon did: He chose (B) — he investigated the .pth file and discovered it was a 34,628-byte credential stealer targeting SSH keys, cloud credentials, Kubernetes configs, and shell history. The machine crash was caused by an unintended fork bomb in the malware. Without the investigation, the community would not have known about the attack. The correct response after investigation: assume all credentials on the machine are compromised and rotate everything — SSH keys, AWS/GCP/Azure credentials, Docker tokens, npm tokens, PyPI tokens, and anything else the malware targeted.

Scenario 3: You are a platform engineer responsible for your company's Python dependency management. After the LiteLLM incident, your CISO asks you to prevent similar supply chain attacks. What is your most impactful single change: (A) pin all dependencies to exact versions, (B) set up an internal package proxy that scans packages before allowing internal use, or (C) require --require-hashes for all pip installs?

The best answer is a combination, but if forced to choose one: (B) — an internal package proxy (like Artifactory or Nexus) provides the broadest protection. It acts as a single control point where packages are scanned, cached, and approved before any internal system can use them. Exact version pinning (A) prevents accidental upgrades but does not protect against a compromised version you have already pinned. Hash verification (C) detects tampering but only works if you have known-good hashes. The proxy gives you all of these plus visibility into what is being used across the organization.

Key Lessons

Pin CI/CD actions to commit SHAs, not tags. Tags like @latest or @v1 can be moved to point at compromised code. Commit SHAs are immutable.
Isolate CI/CD secrets. Scanning tools should never have access to publishing credentials. Use separate jobs with separate permissions.
Use OIDC trusted publishing for PyPI. Eliminates long-lived tokens entirely — no token to steal.
Audit .pth files in your Python environments. The .pth mechanism executes code on every Python startup, no import needed. Most developers do not even know this exists.
A bug in the malware saved the community. The fork bomb was unintentional and led to discovery. Without it, credentials would have been silently stolen for far longer.

Quiz

Q1: How did the attacker get LiteLLM's PyPI publish token? The attacker (TeamPCP) first compromised the Trivy GitHub Action on March 19. When LiteLLM's CI/CD pipeline ran Trivy using an unpinned version, the compromised action exfiltrated the PYPI_PUBLISH token from the GitHub Actions runner environment. The attacker then used that token to publish backdoored versions of LiteLLM to PyPI.

Q2: Why was the .pth file delivery mechanism more dangerous than embedding the payload in a Python source file? A .pth file in site-packages/ executes on every Python interpreter startup — no import litellm required. Every Python script, Jupyter notebook, Django server, and pytest run would trigger the malware. A payload in a .py file would only execute when that specific module is imported.

Q3: Why did the malware accidentally crash the researcher's machine? The .pth payload spawned a new Python subprocess for credential harvesting. But that new subprocess also triggered the .pth execution, which spawned another subprocess, creating an exponential fork bomb. The attacker's code had a bug — it did not guard against recursive execution.

Q4: What types of credentials did the malware target? SSH private keys, .env files, AWS/GCP/Azure credentials, Kubernetes configs, git credentials, Docker Hub tokens, npm tokens, PyPI tokens, shell history (bash/zsh), and in Kubernetes environments, service account tokens from /var/run/secrets/.

Q5: What is OIDC trusted publishing for PyPI and why does it prevent this class of attack? OIDC trusted publishing establishes a trust relationship between GitHub and PyPI using short-lived tokens generated per workflow run. There is no long-lived token to steal — the CI/CD pipeline authenticates to PyPI via GitHub's OIDC identity provider, and each token is scoped to a single publish operation and expires immediately.

One-Liner Summary

A compromised security scanner stole a PyPI token, which was used to backdoor a package with 95 million monthly downloads — and the attack was only discovered because the malware had a bug that accidentally fork-bombed itself.

LiteLLM Supply Chain Attack (March 2026) ​

The Alert ​

Impact ​

Timeline ​

Root Cause ​

The Attack Chain: From Trivy to LiteLLM ​

Two Different Payloads ​

How .pth Files Work ​

The Fork Bomb Bug ​

What Was Stolen ​

The Fix ​

Immediate Response ​

Credential Rotation (MANDATORY if affected) ​

Lessons Learned ​

1. Pin Your CI/CD Dependencies ​

2. Protect CI/CD Secrets ​

3. Use Trusted Publishing for PyPI ​

4. Audit .pth Files ​

5. Monitor for Supply Chain Attacks ​

What You Can Learn ​

For Package Maintainers ​

For Package Consumers ​

For Platform Engineers ​

The Bigger Picture ​

Related Incidents ​

What Would You Do? ​

One-Liner Summary ​

Further Reading ​

Related Pages

LiteLLM Supply Chain Attack (March 2026)

The Alert

Impact

Timeline

Root Cause

The Attack Chain: From Trivy to LiteLLM

Two Different Payloads

How `.pth` Files Work

The Fork Bomb Bug

What Was Stolen

The Fix

Immediate Response

Credential Rotation (MANDATORY if affected)

Lessons Learned

1. Pin Your CI/CD Dependencies

2. Protect CI/CD Secrets

3. Use Trusted Publishing for PyPI

4. Audit `.pth` Files

5. Monitor for Supply Chain Attacks

What You Can Learn

For Package Maintainers

For Package Consumers

For Platform Engineers

The Bigger Picture

Related Incidents

What Would You Do?

One-Liner Summary

Further Reading