--- title: "Filetrace: connecting file reads to network traffic with eBPF" description: "How Filetrace uses eBPF to trace file uploads on Linux: sendfile attribution, read/send heuristics, kernel aggregation, and bounded-memory rankings." author: "Evgenii Orlov" date: 2026-11-17 lastmod: 2026-11-17 canonical: "https://eorlov.org/posts/2026/filetrace-file-to-network-observability/" tags: ["linux","ebpf","observability","systems programming","performance"] --- # Filetrace: connecting file reads to network traffic with eBPF A process is sending a lot of data. You have its PID and the remote address. Which file is it sending? Open files and network counters narrow the search, but they leave you to work out the connection. A Python worker might read its configuration, load several modules, and upload a report through the same thread. If the tool reports all of those files as uploads, you still have to do the investigation yourself. I built [Filetrace](https://github.com/OrlovEvgeny/filetrace) to trace which files a Linux process sends over the network, using eBPF without changing the application. It presents transfers by file, process, and destination, with a confidence level for each association. A long-running agent keeps the collected state available between CLI invocations; `filetrace top` can also collect on its own when there is no agent. The implementation is C23, including the userspace model, CLI, and exporters, with BPF C programs for collection. Linux 6.1 or later on x86-64 or arm64 is the supported collection environment. The macOS build provides the CLI and diagnostics, but does not collect kernel activity. The hard part was deciding when a file read explains a network send, and when the collector should leave the association unknown. That decision also has to fit inside a probe that runs on the application's hot path. I'll walk through both problems using [revision `6e4e257`](https://github.com/OrlovEvgeny/filetrace/tree/6e4e257808191578757077a9c017f8310403fe83). Code excerpts come from that revision; shortened examples are marked explicitly. ## What does it mean to say a file was sent? There are two substantially different observations Filetrace can make: | Observation | Reported method | Confidence | | --- | --- | --- | | The kernel transfers bytes directly from a regular file to a socket | `sendfile` | `EXACT` | | A thread reads a file, then sends bytes on a socket under the correlation rules | `read-write` | `INFERRED` | `EXACT` describes the file-to-socket association. It does not certify that the remote application received or persisted the data. The byte counts are taken at kernel operation boundaries, not from acknowledgements by the receiving application. For ordinary reads and writes, the kernel cannot tell us what happened to a buffer between the two calls. The application may have compressed it, encrypted it, hashed it, or thrown it away. Filetrace does not follow every byte through application memory. Its inference uses thread identity, ordering, byte counts, and the surrounding connection activity. That distinction survives into the data model. Repeated read/send alternation and a close balance between bytes read and sent add evidence, but the result remains `INFERRED`. The classifier does not promote a plausible sequence into an exact observation. The [classification code](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/core/state.c#L925) also applies configurable minimums to ordinary inferred transfers, so a tiny incidental send need not turn into a file-transfer row. A file read may be satisfied from the page cache, so these observations do not measure physical disk traffic. ## Collecting facts and collecting counters The collector has two paths into userspace: ```text Linux hooks and selected userspace library probes | +-- process/socket lifecycle, file paths, names --> BPF ring buffer | +-- network bytes, file-transfer counters -------> active BPF hash map | rotate and drain | userspace state <--------------+ | queries, live UI, exporters ``` A process execution or a newly observed file path is a fact the agent needs to retain. A million sends on the same connection are often useful as counters. Sending one userspace record for each operation would make the collector's transport cost scale with the application's syscall rate, even when the eventual query needs one total. Filetrace therefore accumulates flow and transfer counters in the kernel. The aggregate key identifies the kind of record and the socket/process relationship, with file identity for transfers and destination fields where needed. Byte and operation counters use atomic additions because multiple CPUs can contribute to an entry. The ring buffer carries the descriptive records. Linux's BPF ring buffer shares capacity across CPUs and supports reserving space directly in the buffer, avoiding an intermediate copy for those records. Reservations can fail when the buffer is full; producers do not block waiting for the agent. Those properties are described in the [kernel ring-buffer documentation](https://docs.kernel.org/bpf/ringbuf.html). This split reduces userspace traffic when operations repeatedly hit the same keys. It does not eliminate the per-operation probe cost, and a workload that continually creates new connections or touches new files still creates new state. ## The direct path: `sendfile` The trampoline collector observes the return from `do_splice_direct`, the kernel path used by `sendfile` to move data from a regular file to a socket. At that point it has both file objects and the operation's return value. The [probe](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/probes.bpf.c#L311) starts with these checks: ```c SEC("fexit/do_splice_direct") int BPF_PROG(ft_splice_direct, struct file *in, loff_t *ppos, struct file *out, loff_t *opos, size_t len, unsigned int flags, ssize_t ret) { if (ret <= 0) return 0; struct inode *out_inode = out->f_inode; if ((out_inode->i_mode & S_IFMT) != S_IFSOCK) return 0; struct inode *inode = in->f_inode; if ((inode->i_mode & S_IFMT) != S_IFREG) return 0; /* Remaining body omitted. */ } ``` Using the return value matters. An application can request a large transfer and get a short successful result. Accounting the requested length would overstate what the operation actually moved. The rest of the probe resolves the socket identity and file instance, then adds the returned byte count to a `FT_BPF_AGG_SENDFILE` entry. It also reconciles the transfer with any pending read-based attribution for that file. An application may read a file to calculate a checksum and later send it with `sendfile`; those two observations must not become two uploads. There is a second opportunity for duplication inside the network probes. Pages coming through the direct transfer can also appear at a socket send hook. The trampoline path checks `MSG_SPLICE_PAGES` and excludes those bytes from ordinary read/send correlation while still accounting for network traffic. Network volume and attributed file volume are related measurements with different accounting rules. ## The inferred path: a per-thread byte budget Consider an application loop that reads 16 KiB and sends it, repeatedly. The useful state is small: the file the thread last read, the number of bytes available for attribution, and when that activity happened. Filetrace stores this in `BPF_MAP_TYPE_TASK_STORAGE`. The state belongs to the task, so a numerical thread ID reused later does not inherit an old thread's pending file read. The base accounting rule can be written as follows. This is pseudocode for the rule, with the surrounding heuristics omitted: ```c on_file_read(file, bytes): if file != thread.current_file: reset_pending_read() thread.current_file = file thread.pending += bytes on_socket_send(socket, bytes): expire_stale_read() attributed = min(bytes, thread.pending) account(thread.current_file, socket, attributed) thread.pending -= attributed ``` The pending byte count is a budget. A 4 KiB read cannot explain a subsequent 100 MiB send through this rule. Once attributed, those pending bytes are consumed. Changing the current file resets the foreground state, and stale state expires within the configured correlation window. The BPF configuration starts with a two-second window; ongoing sends update the foreground activity timestamp. The actual [`charge_send`](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/common.bpf.h#L696) helper also accumulates bytes read and the number of read/send alternations. That lets userspace describe why the association was made. The difficult cases begin when a program reads something for its own use immediately before sending unrelated data. ### Background files and files that need stronger evidence A TLS client commonly reads a certificate bundle. A runtime loads modules. A server reads configuration before responding. These reads can sit immediately next to a network operation. Filetrace has configurable `background_paths` for files whose reads should be excluded from this inference. The defaults include certificate stores, resolver configuration, and several runtime metadata files. The agent resolves path patterns to device/inode keys and supplies them to BPF maps; the collector can also learn classifications as file names are observed. Excluding every source file or shared library would hide real transfers of those files. The separate `strict_paths` list handles that ambiguity. Its defaults include `/etc/passwd`, source modules, and shared libraries. When such a file has been read whole, the collector holds the attribution and asks whether the connection's subsequent volume is consistent with sending that file. Incremental streaming has its own treatment. There is a cost to these exclusions: someone can deliberately upload a certificate bundle, and the read-based rule can miss it. I would review the [default patterns](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/agent/config.c#L23) before relying on them for a particular workload. ### Waiting to see what a connection does next Reporting a candidate immediately makes later evidence difficult to use. Filetrace instead holds some charges in bounded windows associated with a socket and thread. A first-flight window covers sends before the connection has heard from its peer. A small initial send followed by a reply and more sending may have been a TLS handshake or a request header. The collector can discard the held file attribution after seeing that continuation. A reply window handles a small file read while servicing an established connection. A configuration read followed by a much larger response should not count as sending the configuration. The implementation compares the window's volume with the bytes of the candidate file and an allowance for headers. The following functions are copied from the [window logic](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/common.bpf.h#L815): ```c static __always_inline bool volume_within(__u64 wire, __u64 whole, __u64 head) { return whole && wire * 10 >= whole * 9 && wire <= whole + head; } static __always_inline bool volume_matches(__u64 wire, __u64 whole) { __u64 head = whole / 2; if (head < CARRY_HEAD_MIN) head = CARRY_HEAD_MIN; if (head > CARRY_HEAD_MAX) head = CARRY_HEAD_MAX; return volume_within(wire, whole, head); } ``` Here the header allowance is clamped between 1 KiB and 2 KiB. The accepted volume starts at 90% of the candidate amount. Despite the variable name `wire`, this is the collector's socket-byte accounting, not a capture of Ethernet frames. Strict whole-file matching adds an allowance for secure framing. The thresholds express a practical assumption about transfers. They can reject a legitimate compressed upload whose size is very different, or accept unrelated traffic of a sufficiently similar size. The confidence label remains necessary after the checks pass. ### Recovering a file read before connection setup Connection setup creates another problem. A client may read a large file into memory, connect, receive handshake data, and only then upload the file. Resetting foreground attribution on receives helps avoid false positives, but can lose this case. The collector retains up to four recent large-file candidates per thread. A newly opened connection gets a bounded candidate set. At settlement, [`dial_judge`](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/common.bpf.h#L1193) requires a unique volume match, a fresh candidate, and sufficiently concentrated sending. It considers the three largest send runs between receives, allowing for interruptions such as TLS 1.3 session tickets. Candidates smaller than 64 KiB do not qualify for this recovery rule. If more than one file fits, it adds no recovered attribution. When one does fit, it adds only the amount the base rule has not already charged. That last subtraction prevents the recovery heuristic from duplicating a partially observed upload. ## Draining counters without losing concurrent increments A shared hash map creates a concurrency problem. Suppose userspace reads an entry and then deletes it. A probe could increment the entry between those operations, and the increment would disappear with the deletion. Filetrace uses two aggregate maps and a one-element `BPF_MAP_TYPE_ARRAY_OF_MAPS` that points to the active one. The agent rotates the outer map to the idle map, then drains the previous map. The kernel's update path waits for pre-existing BPF readers through an RCU grace period, so probes that were using the retired map finish before userspace drains it. This synchronization is explicit in [`maybe_wait_bpf_programs` in Linux 6.1](https://github.com/torvalds/linux/blob/v6.1/kernel/bpf/syscall.c#L157). The rotation in [`drain_counters`](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/loader.c#L1009) begins: ```c __u32 zero = 0; int idle = 1 - b->active; int previous = b->agg_maps[b->active]; if (bpf_map_update_elem(b->agg_fd, &zero, &b->agg_maps[idle], BPF_ANY) != 0) { ft_log(FT_LOG_WARN, "cannot rotate the counter maps: %s", strerror(errno)); return; } b->active = idle; ``` The retired map is read with `bpf_map_lookup_and_delete_batch`, using a default batch size of 512. Flow records are emitted before transfer records from the same drain because transfers refer to the sockets described by those flows. There is also a small idle-time optimization: a per-CPU insertion counter tells the agent whether aggregate entries have appeared since its last flush. When nothing has been inserted, it skips walking an empty map's buckets. Ring-buffer notifications are batched separately. The normal wakeup decision is: ```c static __always_inline __u64 wake_flags(void) { __u64 queued = bpf_ringbuf_query(&events, BPF_RB_AVAIL_DATA); return queued >= cfg.wakeup_bytes ? BPF_RB_FORCE_WAKEUP : BPF_RB_NO_WAKEUP; } ``` A timer provides progress below the byte threshold. Some events need faster attention: observing an executable mapping can trigger userspace probe attachment, and execution events have a separate wakeup path with a minimum gap. Batching is useful until the delay prevents the collector from attaching in time to observe a short-lived process. ## A fallback that does not depend on kernel return probes The preferred collector uses typed `fentry`/`fexit` probes and BTF tracepoints. libbpf's CO-RE relocation support adapts kernel type and field accesses to the running kernel's BTF information. That solves a structure-layout problem; it does not guarantee that every function can accept every probe. The [libbpf overview](https://docs.kernel.org/bpf/libbpf/libbpf_overview.html) explains that portability mechanism. Filetrace tests whether a representative trampoline attachment is usable. In automatic mode, it can select a separate kprobe collector when the trampoline network probes cannot attach. The [fallback](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/fallback.bpf.c) uses entry probes for kernel operations. A kretprobe can hold an instance for the duration of a blocking call; enough blocked calls can exhaust the available instances. Avoiding that dependency requires another way to discover completed work. For TCP sends, the fallback samples `tcp_sock.write_seq` and accounts its advance. For receives, it uses `copied_seq`, adjusting for a consumed FIN. File reads use changes in the task's `ioac.rchar` counter while the collector can still associate that change with the tracked file. The TCP send settlement includes this fragment: ```c __u32 seq = BPF_CORE_READ((struct tcp_sock *)sk, write_seq); __u32 old = t->tx_seq; __u32 delta = seq - old; if (delta == 0 || delta > 0x7fffffff) return; /* A state change can look at the socket at the same moment as its owner. */ if (__sync_val_compare_and_swap(&t->tx_seq, old, seq) != old) return; ``` The compare-and-swap lets only one observer advance a particular baseline. Otherwise, a socket state change and the sending thread could both charge the same sequence delta. The unsigned subtraction accommodates sequence wraparound, while the half-range check rejects implausible backwards deltas. These counters describe bytes accepted into or consumed from the TCP stream, excluding retransmissions and packet overhead. The fallback also has to remember which thread had a claim on the next bytes. That extra state buys compatibility, but gives me a second set of edge cases to test. ## Identities have lifetimes Paths are convenient for display and weak as identities. A file can be renamed, unlinked while open, or replaced at the same path. Inode numbers can eventually be reused. The BPF file cache keys observations by device, inode number, and inode generation. It also tracks the in-memory inode and assigns a Filetrace instance number. The userspace identity is: ```c struct ft_file_id { uint64_t dev; /* stat(2) st_dev encoding */ uint64_t ino; uint32_t instance; /* tells apart successive files that were given the same inode number */ uint32_t pad; }; ``` The inode destruction hook distinguishes a linked file whose in-memory inode is being reclaimed from a deleted file whose number can be assigned to something else. The former can retain its instance; the latter invalidates cached identity and classification state. Path records are emitted when an instance is learned, so repeated reads can use compact identities instead of repeatedly walking and transmitting the path. These rules live in [`file_instance` and `forget_file`](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/common.bpf.h#L1938). The same lifetime problem appears in network and process metadata. The trampoline collector uses socket cookies; the fallback maintains its own socket IDs and destruction handling. Name lookup records belong to a process image, with ancestry information for inherited lookups, rather than a permanent global PID-to-name table. ## Naming the destination and the program An address alone can leave an investigation unfinished. A shared service address may host many names, and a proxy's address identifies an intermediary. Filetrace observes names where supported applications supply them. Uprobes around the C library's `getaddrinfo` capture the requested name and returned addresses. The userspace binding code keeps the process-image context, prefers lookups from the connecting thread, and retains multiple candidates where appropriate. For OpenSSL and GnuTLS, probes capture the server name configured on the TLS object. During a handshake, the name is placed in task-local state and bound to the socket used by the handshake's first observed send. Copying it into that state also avoids looking up the TLS object's name map on every ordinary send. The details are in the [TLS probes](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/bpf/tls.bpf.h) and [name binding code](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/core/binding.c). This can produce a useful description such as `artifacts.example.org via proxy.corp.example`: the lookup names the proxy, while the TLS client asks for the target. It requires no reverse DNS request or TLS payload decryption. These observed names describe application intent; they are not independent proof of a remote service's identity. Coverage depends on the libraries. Go's own resolver and TLS stack, Java, c-ares, rustls, and static binaries do not automatically become visible through these hooks. TLS over memory buffers, including paths used by Node.js and Python's asyncio, can prevent the handshake-to-socket binding. File attribution and destination-name coverage are separate capabilities. Process names need similar context. `python3` identifies an interpreter, not the script being investigated. Filetrace records command lines and working directories and identifies scripts for several interpreters. PHP-FPM and mod_php require request-level attribution because one worker can execute different scripts over its lifetime. PHP probes associate the active request's script with thread state and carry that script identity into transfer records. Attaching a userspace return probe also has consequences for the observed process. The loader includes a [return-probe self-test](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/backends/linux/returns.c), run in a child process with a restrictive seccomp filter, to detect the affected kernel behavior before attaching the dependent PHP, lookup, and TLS probes. Observability code has to account for how its own instrumentation interacts with the workload. ## Ranking files with bounded memory The live model has limits on retained processes, sockets, files, transfers, and metric series. Defaults include 16,384 file entries and 2,048 metric series. Keeping every historical transfer indefinitely would defeat those limits, so the ranking view uses a separate representation. In [`core/leaders.c`](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/core/leaders.c), samples first enter an open five-second slot. When it closes, its summary is folded into rings with different time resolutions and a lifetime summary. Older activity can therefore answer ranking queries without retaining every raw transfer. The slots use Space-Saving tables, with separate rankings by bytes and transfer count. This is an existing streaming algorithm applied to the history problem. A bounded table retains likely heavy hitters while tracking the uncertainty introduced when entries are displaced. The implementation carries byte and transfer error fields into query rows and accounts for the floors of tables where a key was absent. Destination cardinality has a different representation. Small sets retain distinct 32-bit hashes explicitly. Once they exceed the configured small-set capacity, they become HyperLogLog sketches. The per-entry sketch has 128 four-bit registers, occupying 64 bytes; wider aggregation uses 1,024 registers. Combining sketches takes the maximum of corresponding registers. The small representation is exact over the retained hashes, subject to hash collisions; the sketch is approximate. This error is separate from uncertainty about which file was sent. A perfectly counted sequence of inferred transfers is still inferred. These rankings live in the agent's memory. A running agent preserves history between CLI invocations; that is not a promise of durable history across an agent restart. ## Testing what the collector believes The repository separates tests of the userspace system from tests of actual kernel observation. A deterministic simulation backend generates process, socket, and transfer events with a seed and a controllable clock. It drives the state model, queries, and UI without requiring Linux eBPF. The [simulation checks](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/tests/sim/sim_test.sh) exercise both transfer methods and confidence levels, name filtering, process trees, JSON output, and shutdown behavior. This is useful for reproducing a display or accounting bug, but it cannot establish that a real kernel probe sees the right event. The [acceptance suite](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/tests/acceptance/README.md) runs programs that read and send files in controlled sequences. Expected transfers come from what the scenario actually sends. A scenario must detect both missing uploads and files that were incorrectly reported as uploaded. It runs against automatic probe selection and the kprobe path, with explicit outcomes for known limitations, skipped prerequisites, and invalid setups. This is how I approach a change to the inference: reproduce the read/network ordering, state which files the scenario sends, and check both collectors against that expectation. I need to know whether recovering one upload also causes a configuration file to appear in another scenario's results. The acceptance README also records which cases and kernel configurations had actually been exercised at this revision. Some scenarios and the 6.1/6.12 VM harness were still listed as unrun. The existence of a test is not evidence that it passed on every supported kernel. Performance needs equally specific evidence. The [benchmark harness](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/benchmarks/run.sh) compares no agent, minimal collection, and standard collection using small sends, reads, read/send loops, and `sendfile`. It also reports resident memory, idle CPU, and lost events. Minimal mode collects processes, sockets, and network counters; standard mode adds file attribution. I would use those measurements on the target kernel and workload before assigning an overhead percentage. Kernel aggregation explains why the userspace event stream can be smaller. It does not by itself tell us how much slower an application's hot path becomes. ## Missing observations must remain visible The collector can fail to reserve a ring-buffer record, fill an aggregate map, or encounter a probe the kernel cannot attach. It also reads the kernel's per-program `recursion_misses` counters to report executions skipped before the BPF program ran. A program that never ran cannot increment its own loss counter. These conditions affect the interpretation of results. An empty transfer list can mean no observed file transfer, insufficient inference evidence, unsupported I/O, or collection loss. `filetrace doctor` reports usable capabilities and the reasons features are unavailable. Collector statistics expose losses and capacity use alongside the observed traffic. The current collector implements `sendfile` and read/send attribution. The model has names for `splice`, `mmap`, and `io_uring`, but [`ft_method_observed`](https://github.com/OrlovEvgeny/filetrace/blob/6e4e257808191578757077a9c017f8310403fe83/core/model.c#L64) explicitly recognizes only the first two implemented methods. Arbitrary pipe chains, memory-mapped file reads, and asynchronous or cross-thread buffer handoffs cannot be assumed to preserve attribution. Traffic may still be counted even when its source file is unknown. On a supported Linux machine with Filetrace installed, these commands provide a starting point: ```sh sudo filetrace doctor sudo filetrace top sudo filetrace files --json sudo filetrace inspect file /data/report.pdf sudo filetrace connections --remote '*.example*' sudo filetrace events --type file_transfer --since 5m sudo filetrace top --leaders --window 24h ``` The agent also exposes a gRPC API and can export Prometheus metrics and OTLP telemetry. For an investigation, I start with the file and destination views, inspect the method behind a row, and check collection health before treating an absent row as meaningful. The [source and installation instructions](https://github.com/OrlovEvgeny/filetrace) are on GitHub; the [Filetrace documentation](https://filetrace.eorlov.org/) covers configuration and known limitations. For related background, I've written about [Linux futexes](/posts/2023/basics-of-futexes/) and [Bloom filters](/posts/2024/bloom-filters-probabilistic-data-structures/), another case where bounded memory comes with an explicit error model.