BPF based live debugging for Go/C++/Rust in prod with no code changes
Right here is the first in a series of posts describing how we are able to debug applications in production using eBPF, without recompilation/redeployment. This post describes exhaust gobpf and uprobes to make a characteristic argument tracer for Budge applications. This system also can be extendable to assorted compiled languages such as C++, Rust, and heaps others. The next sets of posts in this series will talk about using eBPF for tracing HTTP/gRPC files, SSL, and heaps others.
When debugging, we’re usually drawn to taking pictures the explain of a program. This allows us to query what the utility is doing and judge the keep the trojan horse is found in our code. A easy manner to head attempting to search out explain is to exhaust a debugger to win characteristic arguments. For Budge applications, we continuously exhaust Delve or gdb.
Delve and gdb work smartly for debugging in a vogue environment, but they’re no longer continuously aged in production. The aspects that make these debuggers great may perchance additionally make them undesirable to exhaust in production programs. Debuggers can role off critical interruption to this map and even allow mutation of explain which may perchance additionally lead to surprising mess ups of production tool.
Extended BPF (eBPF) is a kernel technology that is readily accessible in Linux 4.x+. That possibilities are you’ll have confidence of it as a gentle-weight sandboxed VM that runs inner of the Linux kernel and can present verified salvage admission to to kernel memory.
As proven in the overview below, eBPF enables the kernel to lumber BPF bytecode. While the front-halt language aged can vary, it’s continuously a restricted subset of C. In total the C code is first compiled to the BPF bytecode using Clang, then the bytecode is verified to make certain that or no longer it’s protected to carry out. These strict verifications guarantee that the machine code will now not intentionally or unintentionally compromise the Linux kernel, and that the BPF probe will carry out in a bounded preference of instructions every time it’s brought on. These guarantees allow eBPF to be aged in performance-most important workloads love packet filtering, networking monitoring, and heaps others.
Functionally, eBPF enables you to lumber restricted C code upon some match (eg. timer, network match or a characteristic name). When brought on on a characteristic name we name these functions probes and as well they’re going to even be aged to either lumber on a characteristic name inner the kernel (kprobes), or a characteristic name in a userspace program (uprobes). This post focuses on using uprobes to allow dynamic tracing of characteristic arguments.
Uprobes will mean that that you just can intercept a userspace program by inserting a debug trap instruction (
int3 on an x86) that triggers a tender-interrupt . Right here also can be how debuggers work. The float for an uprobe is in actuality the similar as any assorted BPF program and is summarized in the map below. The compiled and verified BPF program is achieved as fraction of a uprobe, and the consequences can even be written correct into a buffer.
Let’s contemplate how uprobes if truth be told characteristic. To deploy uprobes and win characteristic arguments, we are able to be using this easy demo utility. The related elements of this Budge program are proven below.
significant() is a easy HTTP server that exposes a single GET endpoint on /e, which computes Euler’s number (e) using an iterative approximation.
computeE takes in a single seek data from param(iters), which specifies the preference of iterations to lumber for the approximation. The extra iterations, the extra correct the approximation, on the price of compute cycles. It be no longer an important to assign the math in the again of the characteristic. We are factual drawn to tracing the arguments of any invocation of
To assign how uprobes work, let’s gaze at how symbols are tracked inner binaries. Since uprobes work by inserting a debug trap instruction, we want to salvage the address the keep the characteristic is found. Budge binaries on Linux exhaust ELF to store debug data. This data is readily accessible, even in optimized binaries, unless debug files has been stripped. We can exhaust the uncover
objdump to query the symbols in the binary:
From the output, we know that the characteristic
computeE is found at address
0x6609a0. To gaze on the instructions around it, we are able to demand
objdump to disassemble to binary (done by including
-d). The disassembled code looks love:
From this we are able to contemplate what happens when
computeE is known as. The principle instruction is
mov 0x8(%rsp),%rax. This strikes the command offset
0x8 from the
rsp register to the
rax register. Right here is de facto the input argument
iterations above; Budge’s arguments are handed on the stack.
With this data in solutions, we’re now appealing to dive in and write code to mark the arguments for
To win the events, we want to register a uprobe characteristic and occupy a userspace characteristic that can learn the output. A map of here’s proven below. We can write a binary known as
tracer that is guilty for registering the BPF code and reading the consequences of the BPF code. As proven, the uprobe will merely write to a perf-buffer, a linux kernel files structure aged for perf events.
Now that we assign the items alive to, let’s gaze into the significant points of what happens after we add an uprobe. The map below reveals how the binary is modified by the Linux kernel with an uprobe. The tender-interrupt instruction (
int3) is inserted because the first instruction in
significant.computeE. This causes a tender-interrupt, allowing the Linux kernel to carry out our BPF characteristic. We then write the arguments to the perf-buffer, which is asynchronously learn by the tracer.
The BPF characteristic for here’s quite easy; the C code is proven below. We register this characteristic so as that or no longer it’s invoked every time
significant.computeE is known as. Once or no longer it’s known as, we merely learn the characteristic argument and write that the perf buffer. Hundreds boilerplate is required to role up the buffers, and heaps others. and this may perchance even be brand in the total instance here.
Now we occupy a absolutely functioning halt-to-halt argument tracer for the
significant.computeE characteristic! The results of this are proven in the video clip below.
One amongst the icy things is that we are able to if truth be told exhaust GDB to contemplate the adjustments made to the binary. Right here we dump out the instructions on the
0x6609a0 address, sooner than running our tracer binary.
Right here it’s after we lumber the tracer binary. We can clearly contemplate that the first instruction is now
Even supposing we hardcoded the tracer for this explicit instance, or no longer it’s conceivable to make this process generalizable. Many elements of Budge, such as nested pointers, interfaces, channels, and heaps others. make this process worrying, but fixing these considerations enables for one other instrumentation mode no longer readily accessible in present programs. Moreover, since this process works on the binary stage, it would even be aged with natively compiled binaries for assorted languages (C++, Rust, and heaps others.). We factual want to chronicle for the variations of their respective ABI’s.
BPF tracing using uprobes comes with its have role of experts and cons. It be worthwhile to exhaust BPF after we need observability into the binary explain, even when running in environments the keep attaching a debugger will likely be problematic or unpleasant (ex. production binaries). Primarily the most attention-grabbing downside is the code required to salvage even trivial visibility into the utility explain. While BPF code is quite accessible, or no longer it’s advanced to write and take. With out substantial high-stage tooling, or no longer it’s no longer likely this may perchance even be aged for generic debugging.
Budge dynamic logging is one thing we’re working on at Pixie. That possibilities are you’ll checkout this to contemplate how Pixie traces Budge applications running on K8s clusters. If this post’s contents are attention-grabbing, please give Pixie a try, or compare out our delivery positions.