Using eBPF to Safeguard Deployments from Circular Dependencies: A Step-by-Step Guide

By — min read

Introduction

Imagine you’re deploying a critical fix to a broken service, but the deployment script itself relies on that same broken service to download a tool. That’s a circular dependency—a classic deployment nightmare. At GitHub, we faced exactly this problem: our entire source code lives on github.com, meaning if the site goes down, we can’t even access our own code to fix it. Worse, our deployment scripts could inadvertently create new circular dependencies—calling internal services or downloading binaries from GitHub, which might be the very thing that’s failing.

Using eBPF to Safeguard Deployments from Circular Dependencies: A Step-by-Step Guide — Source: github.blog

To solve this, we turned to eBPF (extended Berkeley Packet Filter), a powerful Linux kernel technology that lets you safely run sandboxed programs inside the kernel without changing kernel source code or loading modules. With eBPF, we can monitor every network call and filesystem access made by a deployment script, and selectively block anything that might create a dependency loop. This guide walks you through the same approach: from understanding the types of circular dependencies to writing your own eBPF program that can detect and block dangerous calls during deployments.

What You Need

A Linux system with eBPF support (kernel version 4.9 or later, though 5.4+ is ideal for features like BPF trampoline)
BCC (BPF Compiler Collection) or bpftrace installed (we recommend BCC for fine-grained control)
Basic programming skills in C (for writing eBPF kernel programs) and Python (for user-space tooling)
Understanding of your deployment pipeline—script locations, executed binaries, and expected network/disk activity
Root or sudo privileges (eBPF programs require elevated permissions)

Step 1: Identify Circular Dependencies in Your Deployment

Before writing code, map out all the resources your deployment scripts touch. Consider three categories:

Direct dependencies: The script explicitly downloads a file from github.com (or any internal service) to complete a step.
Hidden dependencies: A tool already on disk checks for an update online before running—if that check times out, the tool may fail.
Transient dependencies: The script calls an internal API (e.g., a migration service) that, in turn, tries to fetch something from the same failing endpoint.

For each deployment script (e.g., a MySQL config change), list all outbound network connections it or its children make. Use strace or tcpdump during a dry run to capture the calls.

Step 2: Set Up Your eBPF Environment

Install BCC from your distribution’s package manager or build from source. For Ubuntu 20.04+:

sudo apt-get install bpfcc-tools linux-headers-$(uname -r)

Verify the installation:

sudo python3 /usr/share/bcc/tools/execsnoop

You should see a stream of new processes being started. This confirms eBPF works on your system.

Step 3: Write an eBPF Program to Monitor Network Calls

Our goal is to attach an eBPF program to the connect() syscall and block any connections to addresses that create circular dependencies. Start with a monitoring-only version. Create a file named block_deploy_conn.c:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("kprobe/tcp_v4_connect")
int trace_connect(struct pt_regs *ctx) {
    // For now, just count and log; we'll block later
    bpf_printk("Outgoing connection detected\n");
    return 0;
}

char _license[] SEC("license") = "GPL";

Compile it with BCC’s Python front end or using clang -O2 -target bpf -c and then load it.

Step 4: Add Blocking Logic for Dangerous Addresses

To block a connection, we need to override the return value of tcp_v4_connect to -EACCES (or inject an error). This requires a BPF trampoline or a raw tracepoint. A simpler approach is to use BPF_PROG_TYPE_CGROUP_SOCK_ADDR with a cgroup where your deployment runs. Attach the program to the connect4 hook inside the deployment’s cgroup.

Here’s a snippet that blocks connections to github.com (IP resolution needed beforehand):

struct sockaddr_in *addr = (struct sockaddr_in *)sk;
u32 dest_ip = addr->sin_addr.s_addr;
// Assume we stored the blocked IP in a map
if (dest_ip == blocked_ip) {
    // Return -EACCES to deny connect
    return 1; // indicates packet should be rejected
}
return 1; // allow

For dynamic IPs, use DNS drop or block by hostname via a user-space helper that populates a BPF map with resolved addresses periodically.

Step 5: Test with a Simulated Circular Dependency

Create a simple deployment script that intentionally tries to download a file from github.com (or your internal service) during a simulated outage (e.g., block your machine’s DNS resolution for that domain). Run the deployment inside the cgroup with the eBPF program attached. Verify that the script fails with a “connection denied” error—proving the circular dependency is broken.

Then test a normal deployment (without the outage) to ensure the eBPF program doesn’t block legitimate connections. Adjust the IP/map filters as needed.

Step 6: Integrate into Your CI/CD Pipeline

Automate the process:

Write a shell script that spawns the deployment within a dedicated cgroup, loads the eBPF blocker at the start, and unloads it upon completion.
Store the blocked-IP list in a version-controlled config file (e.g., YAML) and have the eBPF loader parse it.
Run the script as part of your CI/CD for every deployment to production. If the eBPF denies a call, the pipeline fails early, alerting the team before an outage compounds.

Tips for Success

Start with monitoring, not blocking. Log all connections for a week to identify the real circular dependencies before enforcing blocks.
Use BPF CO-RE (Compile Once, Run Everywhere) for portability across kernel versions. Enable CONFIG_DEBUG_INFO_BTF in your kernel and use BCC’s #include <bpf/bpf_core_read.h>.
Compile resistance into your deployment tools. Make them tolerate temporary outages of non-critical updates—e.g., by adding a timeout that defaults to proceeding if the check fails.
Monitor eBPF performance. eBPF has minimal overhead, but if your deployment forks many processes, ensure your program doesn’t add latency. Use BPF metrics like drop count.
Document your blocked endpoints. Keep a clear mapping of what is blocked and why—useful for onboarding new engineers and auditing.

By following these steps, you’ll have a robust, kernel-level safety net that prevents your deployments from creating new circular dependencies—just like we do at GitHub. Start small, iterate, and let eBPF do the guarding.

Tags: