Using eBPF to Break Circular Dependencies in Deployments: GitHub's Approach

GitHub faces a unique challenge: it hosts its own source code on github.com. This creates a circular dependency where deploying fixes during an outage could require accessing GitHub itself. To mitigate this, GitHub uses eBPF (Extended Berkeley Packet Filter) to monitor and block deployment scripts from inadvertently creating new circular dependencies. Below, we explore how this technology enhances deployment safety through a Q&A format.

What circular dependencies does GitHub face in deployments?

GitHub's deployment system must avoid creating dependencies on the very services it's trying to fix. For instance, if a MySQL outage occurs, deployment scripts might need to pull release data from GitHub—which is unavailable. This is a direct circular dependency. Additionally, hidden dependencies arise when a tool on the machine checks for updates from GitHub, potentially failing or hanging during an outage. Transient dependencies occur when a script calls an internal service (e.g., a migrations service) that then fetches something from GitHub. All three types can prevent successful deployment during incidents, making circular dependency mitigation critical.

Why is a simple code mirror not sufficient?

While GitHub maintains a mirror of its code for fixing forward and built assets for rolling back, this only addresses the basic case of accessing source code. It doesn't solve more subtle circular dependencies introduced by deployment scripts. For example, a script might download a binary from GitHub—bypassing the mirror—or rely on an internal service that itself depends on GitHub. These can't be prevented by code mirrors alone. GitHub needed a more granular solution to monitor and block such calls at the kernel level, which is where eBPF comes in.

What are the three types of circular dependencies?

1. Direct dependency: The deployment script itself attempts to fetch something from GitHub (e.g., an open source tool). During an outage, this fails.

2. Hidden dependency: A locally present tool checks for updates online. When GitHub is down, the check may hang or error out, blocking the script.

3. Transient dependency: The script calls an internal API, which in turn makes a request to GitHub. The failure propagates back. Identifying these dependencies is tricky because they may not be obvious from the script's code. Prior to eBPF, each team had to manually review their deployment scripts—a process that was often incomplete.

How does eBPF help solve these dependencies?

eBPF allows GitHub to selectively monitor and block system calls made by deployment scripts. By writing eBPF programs that attach to kernel hooks, GitHub can detect when a script attempts to access network resources (like contacting GitHub) and take action—either logging the event or blocking the call entirely. This provides a safety net that stops deployment code from introducing circular dependencies automatically, without requiring manual review of every script. eBPF works at the kernel level, so it can intercept any type of dependency, whether direct or indirect.

What did GitHub find from their eBPF evaluation?

When designing their new host-based deployment system, GitHub evaluated eBPF as a way to prevent circular dependencies. They found that eBPF could effectively block unauthorized network calls from deployment scripts, while allowing legitimate traffic. For example, they could write programs that detect attempts to access specific hosts (like github.com) and deny them during deployment, ensuring that scripts rely only on local resources. The evaluation showed that eBPF introduced minimal performance overhead and was flexible enough to adapt to different dependency patterns. This led GitHub to adopt eBPF as part of their deployment safety toolkit, reducing the risk of cascading failures.

How can others get started with eBPF for deployment safety?

To replicate GitHub's approach, start by learning eBPF basics through resources like the official documentation or books such as “Linux Observability with BPF.” You'll need to write eBPF programs (often in C or using higher-level frameworks like BCC) that attach to tracepoints or kprobes related to networking. For deployment safety, focus on connect() or sendto() system calls to monitor outbound connections. Use maps to store allowed/blocked destinations. Test on non-production systems first. Remember: eBPF can block calls, so validate your policies carefully. Start small—monitor without blocking—then gradually add restrictions. Tools like bpftrace can help prototype quickly.