At GitHub, we dogfood our own platform—meaning all our source code lives on github.com. This gives us the advantage of testing changes internally before rolling them out to users. However, it also creates a unique vulnerability: if github.com ever goes down, we cannot access the very code needed to fix it. This circular dependency can ripple through our deployment scripts, creating failure points that are hard to anticipate. Below, we explore how we identified these patterns and implemented a novel solution using eBPF to keep deployments safe and resilient.

What is the core deployment challenge GitHub faces due to hosting its own code?

GitHub's reliance on its own infrastructure creates a circular dependency: to deploy new code, we need GitHub to be operational, but if GitHub is down, we can't push fixes. This forces us to maintain a mirror of our code for fixing forward and built assets for rolling back. However, the problem doesn't end there. Deployment scripts often have their own dependencies—pulling binaries, checking for updates, or calling internal APIs—that can introduce further circular chains. When an incident like a MySQL outage strikes, these scripts may fail because they depend on a service that is now unavailable, turning a simple fix into a cascading crisis.

How does a simple circular dependency manifest during a MySQL outage?

Imagine a MySQL outage makes GitHub unable to serve release data from repositories. To resolve it, we need to run a deploy script on each affected node to roll out a configuration change. The script attempts to pull the latest open source tool from GitHub—a direct dependency. Since GitHub is down, the script cannot complete. This direct dependency is the most obvious form, but other types lurk beneath the surface. For instance, a hidden dependency occurs when a tool already on the disk checks for an update and hangs because GitHub is unreachable. A transient dependency arises when the script calls an internal service that, in turn, tries to fetch a binary from GitHub, propagating the failure. Each type adds a layer of fragility.

What are the three types of circular dependencies GitHub identifies?

GitHub classifies circular dependencies into three categories:

• Direct dependencies: The deploy script itself attempts to download a resource from GitHub (e.g., a binary, release data). If GitHub is unavailable, the script fails immediately.
• Hidden dependencies: Tools or services already present on the host check for updates, call home, or validate licenses by contacting GitHub. These checks may not be obvious to script authors and can cause timeouts or errors.
• Transient dependencies: The script calls an internal API (e.g., a migrations service), which then tries to fetch something from GitHub. The failure cascades back, making the script's root cause hard to diagnose.

Understanding these types helped us design a system that can catch them all without requiring developers to manually audit every call.

How does eBPF help prevent circular dependencies in deployment scripts?

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows sandboxed programs to run in kernel space. GitHub uses eBPF to monitor and selectively block network calls made by deployment scripts. We attach eBPF programs to system calls like connect(), sendto(), and open(). When a script tries to reach an IP address or domain that could create a circular dependency (e.g., github.com or an internal service that depends on GitHub), the eBPF program intercepts the call. If the destination is on our blocklist—dynamically updated based on incident state—the call is denied or logged. This gives us a runtime safety net: even if a script contains a hidden dependency, eBPF prevents it from completing the bad request.

Can you explain how eBPF monitors and blocks unwanted network calls?

We deploy eBPF programs that hook into kernel functions such as tcp_connect() and udp_sendmsg(). These programs run with minimal overhead and inspect the destination IP and port. Our control plane pushes a set of circular dependency rules to each host. For example, during an incident, we might add rules to block any outbound TCP connection to github.com on port 443. The eBPF program checks each outgoing connection attempt against these rules. If a match is found, it either drops the packet entirely or returns an error code (e.g., ECONNREFUSED) to the calling process. The deployment script sees the failure and can handle it gracefully—often by falling back to a local cache or skipping the update. This approach is both low-latency and secure, as the eBPF bytecode is verified and runs in a sandbox.

What makes eBPF a better solution than traditional dependency checks?

Traditionally, teams had to manually review deployment scripts and identify each possible circular dependency—a burdensome and error-prone task. Scripts evolve, tools update, and new dependencies appear. eBPF offers a runtime, kernel-level enforcement that doesn’t require modifying the script or the application code. It can catch dependencies that are not explicitly listed—like a tool phoning home for telemetry—because it observes actual system calls. Moreover, eBPF programs can be updated dynamically: we can push new blocking rules during an incident without rebooting hosts or restarting services. This makes the solution both comprehensive and agile, reducing the toil on operations teams while improving deployment reliability.

How can teams start writing their own eBPF programs for deployment safety?

Getting started with eBPF requires a Linux environment (kernel 4.18+) and the bpf toolchain (clang, llvm, libbpf). We recommend the bcc framework for simpler prototyping or libbpf-based programs for production. Begin by writing a minimal program that attaches to syscall:connect and prints the destination IP. You can extend it with a hash map to store blocked addresses. Use the bpf() syscall to load the program and attach it via a raw tracepoint or kprobe. For a more structured approach, explore the eBPF documentation and GitHub’s own open-source tools. Once your program works, deploy it as a daemon that syncs rules from a central store. Remember to test thoroughly in a staging environment—eBPF is powerful but running it in kernel space demands care.