Securing Deployments with eBPF: A Step-by-Step Guide to Preventing Circular Dependencies

Introduction

Imagine your deployment script silently creates a circular dependency—pulling a binary from the very service you're trying to fix. This nightmare scenario is all too real for companies hosting their own infrastructure. At GitHub, we discovered that even a simple MySQL outage could cascade into a deployment failure if scripts rely on GitHub itself to fetch tools. To break this loop, we turned to eBPF (extended Berkeley Packet Filter).

Securing Deployments with eBPF: A Step-by-Step Guide to Preventing Circular Dependencies — Source: github.blog

In this guide, you'll learn how to use eBPF to monitor and block system calls that introduce circular dependencies during deployments. By the end, you'll be able to proactively prevent your deployment pipeline from creating self-referential traps—without modifying your existing scripts.

What You Need

Linux kernel version 4.18 or newer (eBPF support is required)
Development tools: clang, llvm, libbpf-dev, and bpftool
Python 3.6+ (for helper scripts, optional)
Access to deployment hosts with root or sudo privileges (for loading eBPF programs)
Basic knowledge of C programming (to write eBPF programs) and system calls
Your deployment scripts and a list of external dependencies (e.g., GitHub releases, internal APIs)

Step-by-Step Guide

Step 1: Identify Your Circular Dependencies

Before you can block them, you need to know what you're up against. Review the three types of circular dependencies we encountered:

Direct dependencies: Your deployment script downloads a tool from the very service it's trying to fix (e.g., fetching a binary from GitHub during a GitHub outage).
Hidden dependencies: A local tool, already on disk, checks for updates by contacting the same service.
Transient dependencies: Your script calls an internal API, which in turn tries to download something from the outside service.

Audit your deployment scripts and any tools they invoke. List every external host, API endpoint, or package repository that could become unavailable during an incident.

Step 2: Set Up Your eBPF Development Environment

Install the necessary tools on a development machine or directly on a test host:

sudo apt-get update
sudo apt-get install -y clang llvm libbpf-dev linux-tools-$(uname -r)

Verify your kernel supports eBPF:

bpftool feature probe

You should see output confirming that eBPF programs and maps are available.

Step 3: Write a Basic eBPF Program to Monitor Network Calls

Start by writing a simple eBPF program that hooks into the connect system call. This will let you see every outgoing network connection made by your deployment process. Save the following as monitor_connect.c:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <linux/ip.h>
#include <linux/tcp.h>

SEC("kprobe/connect")
int trace_connect(struct pt_regs *ctx)
{
    bpf_printk("connect syscall triggered\n");
    return 0;
}

char _license[] SEC("license") = "GPL";

Compile it:

clang -O2 -target bpf -c monitor_connect.c -o monitor_connect.o

Load and attach the program to a specific process (e.g., your deployment script's PID).

Step 4: Enhance the eBPF Program to Block Specific Destinations

Now modify the program to inspect the destination IP address and block connections to services you have identified as dangerous (e.g., GitHub API). Use an eBPF map to store the blocklist dynamically. Example (partial):

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, __u32);       // IP address
    __type(value, __u8);      // 1 = block, 0 = allow
} blocked_ips SEC(".maps");

SEC("kprobe/tcp_v4_connect")
int block_connect(struct pt_regs *ctx)
{
    // Retrieve socket structure and extract destination IP
    // Compare against blocked_ips map
    // If match found, return -1 to block
    return 0;
}

Important: Blocking system calls can break legitimate functionality. Test thoroughly in a sandbox first.

Step 5: Test the eBPF Program on a Staging Environment

Deploy your eBPF program to a staging host that mirrors your production setup. Run your deployment scripts with the eBPF program enabled in monitor-only mode (i.e., log but don't block). Analyze logs to ensure you haven't accidentally flagged necessary connections.

Once satisfied, switch to blocking mode and re-run the deployment. Verify that the script fails gracefully (e.g., uses a local cache or falls back to an alternative) rather than hanging.

Step 6: Deploy to Production with Rollback Procedures

Integrate your eBPF program into your deployment pipeline. Use a tool like bpftool to load the program before the deployment script starts and detach it after completion. Add safety measures:

Automatic rollback: If the eBPF program itself fails to load, abort the deployment.
Graceful exit: Ensure deployment scripts have a timeout and fallback path.
Monitoring: Log every blocked call for later review.

Tips for Success

Start with monitoring only – blocking prematurely can cause more outages. Let the eBPF program log all dependencies first.
Use eBPF maps for dynamic rules – update the blocklist without recompiling the program by writing to map entries from userspace.
Test every dependency – remember hidden and transient dependencies. Use the eBPF logs to build a complete picture.
Combine with other tools – eBPF is powerful, but also consider network policies (e.g., iptables) and service meshes for defense in depth.
Keep the kernel updated – newer kernels have better eBPF features and security guarantees.

By following these steps, you can eliminate the risk of circular dependencies in your deployment pipeline. eBPF gives you surgical control over what your scripts can access—even when everything else is failing.