Inside the Shai-Hulud Attack: A Guide to Detecting and Defending Against the Lightning PyPI Supply Chain Compromise

Overview

In late April 2026, the PyPI ecosystem faced a sophisticated supply chain attack targeting the popular deep learning library lightning (the official PyTorch Lightning package). Versions 2.6.2 and 2.6.3, published on April 30, 2026, were compromised with a hidden JavaScript payload designed to steal credentials, cloud secrets, and token authentication data, while also propagating itself across the npm ecosystem. This guide provides a technical deep dive into the attack mechanism, detection steps, remediation actions, and defensive strategies. Whether you are a data scientist, MLOps engineer, or security professional, understanding this threat is critical to protecting your AI/ML pipelines.

Prerequisites

To follow this guide effectively, you should have:

• Basic familiarity with Python package management (pip, requirements.txt)
• Access to a development or production environment where `lightning` may be installed
• Optional: A Semgrep account for automated scanning (the team has released a dedicated rule)
• Knowledge of GitHub token rotation and cloud credential management

Step-by-Step Instructions

1. Identify if You Have Installed an Affected Version

The malicious versions are lightning==2.6.2 and lightning==2.6.3. Run the following command to check your environment:

pip show lightning | grep -E '^Version: (2\.6\.2|2\.6\.3)$'

If this returns a match, proceed immediately to Remediation. Even if you are not directly using `lightning`, it may be a transitive dependency of other packages. Use:

pipdeptree -p lightning

to see if it appears in your dependency tree.

Note: The compromise occurred on April 30, 2026. Anyone who ran pip install lightning on or after that date may be affected.

2. Scan Your Project with Semgrep (Recommended)

The Semgrep team has published an advisory and a specific rule to detect the malicious payload. If you have a Semgrep account, perform the following steps:

1. Log into your Semgrep account and navigate to the Advisories page.
2. Look for the advisory related to lightning (PyPI) – Shai-Hulud.
3. Trigger a new scan on your projects if you haven't recently.
4. Check the dependency filter for matches. If you see “No matching dependencies,” you are safe. If you see a match, note the affected projects and continue to the next steps.

If you don’t have a Semgrep account, you can manually inspect your project for the indicators of compromise (IOCs) listed below.

3. Manual Inspection for Indicators of Compromise (IOCs)

The malware installs a hidden _runtime directory inside the `lightning` package folder. This directory contains obfuscated JavaScript files that execute automatically upon module import. Additionally, the worm creates suspicious directories in your home or project environment:

• ~/.claude/ (unexpected content)
• ~/.vscode/ (unexpected content)
• Public GitHub repositories named EveryBoiWeBuildIsaWormBoi

To check for the _runtime directory, locate the installation path:

python -c "import lightning; print(lightning.__file__)"

Then navigate to the parent directory and look for a _runtime folder:

ls -la /path/to/site-packages/lightning/ | grep _runtime

If present, the package is compromised. Also scan your GitHub repository list for unusual repository names matching the worm pattern.

4. Remediation Actions (If Affected)

Immediately take the following steps in order:

1. Remove the malicious package:

   pip uninstall lightning

   Then reinstall a clean version (downgrade to e.g., 2.6.1 or upgrade to 2.6.4 if released):

   pip install lightning==2.6.1

2. Rotate all credentials: This includes GitHub tokens, cloud provider API keys (AWS, GCP, Azure), and any environment variables containing authentication secrets. The malware exfiltrates data via four parallel channels (HTTPS POST to C2 server). Assume all secrets exposed.
3. Audit your repositories: Use the GitHub API or a local clone to check for any injected files in .claude/ and .vscode/ directories. If found, delete them and review recent commits for malicious code.
4. Check npm packages: Since the worm propagates from PyPI to npm, if you have npm publish credentials on the same machine, it may have injected a setup.mjs dropper and router_runtime.js into packages you maintain. Run npm audit and inspect package.json scripts for a preinstall hook pointing to setup.mjs. If present, remove the hook and delete the files, then rotate those npm tokens.
5. Scan your CI/CD pipelines: Many developers run `pip install lightning` in CI. Check logs for the affected dates and credential usage.

5. Long-Term Prevention and Monitoring

To avoid falling victim to similar attacks in the future, implement these practices:

• Pin your dependencies: Use exact versions in requirements.txt or pyproject.toml. Avoid floating ranges like >=2.0.
• Use package integrity verification: Leverage pip hash or tools like pyproject.toml hashes.
• Monitor for suspicious package updates: Set up alerts for new releases of critical dependencies and review their changes before updating.
• Adopt dependency scanning tools: Semgrep, Dependabot, or Snyk can detect known malicious packages quickly.
• Isolate build environments: Use containers or virtual machines for builds, and limit inter-environment credential sharing.

Common Mistakes

Even experienced developers can fall into these traps:

• Ignoring minor or patch version bumps: Attackers often target patch versions to appear harmless. Always review changelogs and diffs.
• Not rotating tokens after a potential exposure: Even if no IoCs are found, if your environment was active during the attack window, rotate keys as a precaution.
• Assuming only direct dependencies matter: Transitive dependencies are a huge surface. Always run pip freeze to see the full tree.
• Forgetting to check development machines: Many credentials are stored in local environment files. The malware can steal from any machine that imports the package.
• Ignoring npm warnings: If you receive alerts about suspicious package behavior after the attack (e.g., preinstall scripts), investigate immediately.

Summary

The Shai-Hulud malware campaign marks a significant escalation in AI/ML supply chain attacks, leveraging the trust in PyTorch Lightning to deploy a cross-ecosystem worm. By understanding the attack vectors, running the detection steps outlined above, and proactively rotating credentials, you can mitigate the damage. The key takeaways are: check your `lightning` version, scan for IoCs, rotate all secrets, and tighten your dependency management. Stay vigilant—similar attacks are likely to emerge.