Self-Validating Uncertainty

How to do measurement when you cannot afford ground truth.

Last updated on Nov 2, 2026 Research

Project Overview

Internet measurement often has to infer what it cannot directly observe: locations, dependencies, physical routes, failures. Under this kind of partial observability, different realities can fit the same evidence, and ground truth is scarce, selective, or inaccessible — usually scarcest exactly where confidence is most needed. Yet validation remains tied to correctness: a claim is trusted only to the extent that it can be individually verified.

We argue that this creates a structural bottleneck. When correctness cannot be established at scale, validation should shift from correctness to calibration — whether a system’s stated confidence reflects its error. That requires outputs that carry auditable descriptions of their own reliability. Uncertainty must sometimes also be typed by its source: a candidate set may reflect genuine multiplicity in the world or merely insufficient evidence, and the distinction determines how the result can be validated.

Self-validating uncertainty does not overcome missing evidence. It makes the boundaries of what we can claim visible.

Explore the evidence: Interactive HotNets ‘26 artifact

Case Study: Inferred Internet-Path Geolocation

We test the idea on roughly 408,000 inferred Internet paths, of which only 4,678 (about 1.1%) have reference locations from active validation — and that pool is not a random sample, since shorter paths are easier to validate. Every path carries ℛ, a label-free path–model alignment score. ℛ is not a probability of correctness; instead, scarce labels are used to estimate how often paths in each ℛ group are correct, with uncertainty intervals.

A ground-truth-withholding experiment shows the resulting validation dividend: as labels are removed, correctness-based validation collapses in proportion to the labels available, while group-level reliability estimation degrades far more gracefully. The same analysis also shows its limits — a precise estimate in the labeled pool does not by itself establish that it transfers to the corpus.

The interactive artifact lets you vary the label budget and sampling regime, inspect where labels land in individual simulation draws, and examine whether the labeled sample can speak for the corpus. It is built directly from the experimental exports.

Publications