When your entire codebase lives on your own platform, a single outage can create a paradox: to fix the platform, you need the platform. That's the circular dependency nightmare GitHub faced. Traditional mitigation—like maintaining a mirror—only scratched the surface. In designing a new host-based deployment system, GitHub turned to eBPF (extended Berkeley Packet Filter) to intelligently monitor and block dangerous calls in real time. Below, we break down the problem and the eBPF-powered solution into eight key insights.
1. The Core Circular Dependency Problem
GitHub hosts all its source code on github.com—a fact that makes them their own biggest customer. This dogfooding ensures internal stability before features reach users, but it introduces a glaring risk: if github.com goes down, the very code needed to fix it becomes inaccessible. To break this loop, GitHub maintains a mirror of source code for emergency fixes and pre-built assets for rollbacks. Yet this is only a partial fix—deployment scripts themselves can introduce new circular dependencies. That's where eBPF enters the picture.
2. Direct Dependencies: When the Fix Relies on the Broken System
Imagine a MySQL outage that stops GitHub from serving release data. To resolve it, you must run a deploy script on the affected MySQL nodes. A direct dependency occurs when that script tries to pull the latest open-source tool from GitHub itself. Since GitHub is down, the script fails—you're stuck. eBPF can intercept such outbound calls during deployment and block them if they target the failing service, instead redirecting to a local cache or mirror.
3. Hidden Dependencies: The Silent Saboteurs
Deploy scripts often use servicing tools already present on disk. A hidden dependency surfaces when that tool, upon execution, checks GitHub for an update. During an outage, the check may hang or throw an error, stalling the entire deployment. These dependencies are hard to spot in code reviews. eBPF, running at the kernel level, can monitor all system calls made by the tool and block any that attempt to reach github.com, forcing the tool to proceed offline.
4. Transient Dependencies: Chain Reactions That Break Deployments
A transient dependency involves a cascade: the deploy script calls an internal service (e.g., a migration service) via API, which in turn tries to fetch a binary from GitHub. The failure propagates back to the script. This indirect reliance is nearly impossible to document or predict. eBPF's ability to trace inter-process communication and network calls allows GitHub to blacklist any chain that ends up touching the downed service, even if it's two hops away.
5. The Old Way: Manual Code Reviews Fall Short
Before eBPF, the burden fell on individual teams to inspect their deployment scripts for circular dependencies. This approach was ad hoc and error-prone. Hidden and transient dependencies easily slipped through, and review processes couldn't keep up with rapidly changing scripts. Moreover, a script that worked fine during normal operations could suddenly break under outage conditions. Clearly, a runtime solution was needed—one that didn't rely on human vigilance.
6. Enter eBPF: A Kernel-Level Safety Net
eBPF allows safe, low-overhead injection of programs into the Linux kernel. GitHub saw an opportunity: by writing eBPF programs that hook into system calls and network events, they could selectively monitor deployment traffic. If a process tries to connect to a blocked endpoint (like github.com during an outage), the eBPF program can immediately deny the call. This happens without modifying the deployment scripts or waiting for code reviews, providing a dynamic safety net.
7. How eBPF Monitors and Blocks Dangerous Calls
GitHub's eBPF programs run on every host involved in deployments. They track all outbound connections and exec system calls, maintaining a whitelist of allowed targets. For example, during a MySQL outage, the eBPF program can block any TCP connection to the GitHub release endpoint. It can also intercept attempts to download binaries or check for updates. The result: deployment scripts fail fast if they try to create a circular dependency, with clear error logs that help developers fix the root cause.
8. Getting Started with eBPF for Deployment Safety
To implement a similar system, start by identifying your critical circular dependencies. Then write a simple eBPF program using BCC or bpftrace that hooks into connect() syscalls. Filter by destination IP or port. Test in a sandboxed environment, then deploy to production with a monitor-only mode first. Gradually escalate to blocking once you're confident. GitHub's approach proves that with eBPF, you can enforce deployment safety without sacrificing speed.
eBPF transforms how we handle runtime dependencies. By shifting from manual reviews to automatic kernel-level enforcement, GitHub has made its deployments resilient to the very outages they're meant to fix. Whether you're running a small cluster or a global infrastructure, eBPF offers a lightweight, powerful tool to break circular dependencies—one syscall at a time.