CS 445: Internet-Scale Experimentation | Aqualab - Northwestern University

Overview

Mon, 01 Jan 0001 00:00:00 +0000

Syllabus Calendar & Readings Course Project Reading, Reviews & Presentations

How would you …

Evaluate the effectiveness of a new feature in your startup’s app?
Understand the tense dynamics between content providers and ISPs?
Analyze the impact of COVID-19 on Internet infrastructure?
Assess what drives Internet users’ quality of experience?
Distinguish between large-scale failures and deliberate Internet shutdowns?

These seemingly diverse questions share a common thread: the need for experimentation at Internet scale.

Internet-Scale Experimentation is a graduate-level course that blends an introduction to the field with a seminar format, offering a comprehensive exploration of the unique challenges of conducting large-scale experimentation and measurement in networked systems.

Over the past few decades, networked systems have evolved into essential components of daily life and critical foundations of economic, educational, health, and defense systems. While this progress highlights the success and societal impact of our field, it also introduces significant complications for experimentalists. Conducting randomized controlled experiments at even fractions of the scale of today’s systems is nearly impossible. Yet, as we venture into uncharted territories with new ideas, we are increasingly expected to provide compelling evidence of the effects of interventions.

This course equips you with tools and methodologies to meet these challenges. By combining foundational knowledge with hands-on learning, you will:

Engage with leading research by reading, reviewing, and presenting academic papers.
Participate in discussions on real-world challenges and solutions for Internet-scale experimentation.
Conduct your own research project(s), leveraging existing platforms and datasets to investigate critical questions.

This hybrid introduction and seminar structure offers both foundational insights into Internet-scale experimentation and a platform to grapple with its complexities. Through this approach, you’ll develop the skills to design and evaluate impactful experiments in an increasingly networked world.

Topics include

Strategies for sound Internet measurements
Experimental design and methodologies
Measurement tools and platforms
Network infrastructure and traffic analysis
Applications and distributed services: DNS, Web, CDNs, Video, LEO satellite networks, AI services, …
Privacy, security, and policy implications

Course Organization

The course is structured into two main parts:

A five-week introduction to foundational concepts and methods.
A series of in-depth paper discussions and project work.

The project component includes:

A warm-up project to build essential skills.
A reproducibility project to apply and deepen your understanding by reproducing results from a published paper.

Most class meetings are centered around paper presentations and discussions. You should read each paper before coming to class and be prepared to discuss it.

What you get out of the course depends on what you put into it—this is especially true for seminars. You should read all papers tagged as Background, Question, and Review before coming to class and be prepared to engage. As a reading enforcer, before each class you must answer (in Canvas) a short question for all required readings (i.e., papers tagged as Question or Review) for which you are not the assigned reviewer.

The class will run as a mini-conference with you as Program Committee members. We will use a subset of papers in the schedule (those tagged Review) as our “submissions.” Each of you will write reviews for several of them. We will hold a PC-style meeting at the end of the quarter to decide which papers “should be accepted” for publication.

While there is no required textbook, a classic reference is:

M. Crovella and B. Krishnamurthy, Internet Measurement: Infrastructure, Traffic and Applications, Wiley 2006.

Communication Channels

The course web site (this Hugo site + Canvas) will carry announcements, schedule updates, and corrections. Check regularly.
Piazza (linked from Canvas) will be used for general discussions and to help identify projects and groups.
Canvas and email may be used for class discussion and logistics.
For email to course staff, please start your subject line with:
ISE: <short-topic>
to ensure a prompt response.

Grading

Grading is criterion-referenced, not curved: your grade depends on how well you do relative to predefined performance levels, not how others perform.

If a test has 100 possible points, for example, then:

≥ 90 → in the A range (90–92: A-, etc.)
≥ 80 → in the B range
≥ 70 → in the C range

If everyone works hard and earns ≥ 90, everyone gets an A.

Total scores (0–100) are determined roughly as:

Paper discussion participation (and questions): 10%
Paper reviews: 20%
PC meeting participation: 5%
Paper presentation: 20%
Warm-up project: 15%
Reproducibility project: 30%

Calendar

Mon, 01 Jan 0001 00:00:00 +0000

Syllabus Calendar & Readings Course Project Reading, Reviews & Presentations

Required readings are tagged as Background , Question (short pre-class answer on Canvas), or Review (WINE 2027 submission).

This schedule may change slightly as the quarter proceeds. When a meeting has two readings, the first generally introduces the instrument, dataset, or research question, and the second offers a comparison or a reason to revisit the assumptions of the first. Each card’s note says what the meeting adds to the course; the Discussion line is the question we will work through in class.

Week 1

Tue 1/5 Introduction to Internet-scale experimentation

Background

How to Read a Paper, S. Keshav, SIGCOMM CCR 2007

Strategies for Sound Internet Measurements, V. Paxson, IMC 2004

Discussion. What would it take for you to trust a surprising measurement result?

Presenter: Fabián.

Sets up the two habits the course builds on: reading research efficiently and critically, and treating every measurement as the output of an instrument whose behavior you must understand.

Thu 1/7 What the Internet looks like in practice

Background

Internet Inter-Domain Traffic, C. Labovitz et al., SIGCOMM 2010 (selected sections)

Question

Seven Years in the Life of Hypergiants’ Off-Nets, P. Gigis et al., SIGCOMM 2021

Discussion. Which networks carry the traffic, and where is content actually served from? How did the authors observe off-net servers from the outside, and what would that method miss?

Presenter: Fabián.

Before measuring the Internet you need a model of what is out there. This meeting updates the textbook picture: traffic has concentrated in a few hypergiants, which now place their servers deep inside other operators' networks.

Week 2

Tue 1/12 Measurement tools

Background

Avoiding Traceroute Anomalies with Paris Traceroute, B. Augustin et al., IMC 2006

Question

TTL Jumps: Unexpected TTL Rewrites Impacting Inferences from Traceroutes, S. Kappes, A. Feldmann, T. Fiebig, J. Zirngibl — IMC 2026

Alt: RemapRoute: Local Remapping of Internet Path Changes, E. Fazzion et al., IMC 2025 (short)

Discussion. When a traceroute shows a link between two routers, what assumptions let us read it as a real adjacency? What evidence would distinguish a load-balancing artifact from hops hidden by TTL rewriting?

Presenter: Fabián.

Every inference inherits the assumptions of its instrument. Using traceroute as the case, learn to name those assumptions and recognize the failure modes that violate them.

Thu 1/14 Measurement platforms

Background

A Survey on Internet Performance Measurement Platforms and Related Standardization Efforts, V. Bajpai and J. Schönwälder, IEEE Communications Surveys & Tutorials 2015 (selected sections)

Review

Rooting Out Incorrect RIPE Atlas Probe Geolocations, K. Izhikevich, B. Du, S. Rao, M. Tran, A. Ukani, R. Bellis, L. Izhikevich — IMC 2026

Alt: Censys: A Map of Internet Hosts and Services, Z. Durumeric et al., SIGCOMM 2025

Discussion. Who supplies a probe’s location, and how could you check it? The slides also cover Censys as a contrasting platform model, where researchers query results of Internet-wide scans.

Presenter: Fabián.

Platforms determine where you can measure from and what you can see. Learn to compare them and to check the metadata you rely on; this prepares you directly for the router-geolocation warm-up.

Week 3

Tue 1/19 Collecting and combining observational data

Review

Understanding Different Perspectives: Analyzing UDP Scanning Through Reactive Telescopes, D. Ferrero, A. Sordello, H. Griffioen, I. Drago, G. Smaragdakis, M. Mellia — IMC 2026 (short)

Background

The Wisdom of the Measurement Crowd: Building the Internet Yellow Pages, a Knowledge Graph for the Internet, R. Fontugne et al., IMC 2024

Alt: Lessons Learned from Operating a Large Network Telescope, A. Männel et al., SIGCOMM 2025

Alt (companion on sensor placement): Lightscope: Turning Production Machines into Telescopes and Honeypots, E. Kapitanski, J. Mirkovic, IMC 2026

Alt (for Yellow Pages): How Did You Find That Address? An In-Depth Investigation of Domain-Based IPv6 Scanners, S. Kappes et al., CoNEXT 2026 (PACMNET)

Discussion. What becomes visible only after the instrument replies? What do you inherit, and what must you check, when you build on datasets collected by others?

Presenter: Fabián.

Not every study needs new probes. Learn how passive, reactive, and reused data each shape what you can observe, and what you take on when you combine measurements others collected.

Thu 1/21 Natural and quasi-experiments

Background

Conceptualizing Natural and Quasi-Experiments in Public Health, F. de Vocht et al., BMC Medical Research Methodology 2021 (selected sections)

Question

Need, Want, Can Afford: Broadband Markets and the Behavior of Users, Z. S. Bischof, F. E. Bustamante, R. Stanojevic — IMC 2014

Optional Measuring the Impact of RPKI on the BGP Updates Volume, S. Quinzi, C. Pelsser, G. Di Battista — CoNEXT 2026 (PACMNET)

Discussion. What comparison would tell us whether access capacity changed demand? State it in your own words and name one plausible competing explanation. For Quinzi et al.: how much update activity would have happened anyway, and what baseline would let you estimate it?

Presenter: Fabián.

Randomized experiments are rarely possible at Internet scale. Natural and quasi-experiments are the course's core tool for causal claims: finding a comparison that stands in for the experiment you cannot run.

Week 4

Tue 1/26 From a sample to a population

Question

A Global Inference and Assessment of Large Shared IP Addresses, V. Giotsas, L. Salamatian, A. Cordelle, N. Wood, M. Fayed — SIGCOMM 2026

Review

HERMES: Repurposing User-Driven Speed Tests to Monitor the Internet, L. Salamatian, K. Vermeulen, D. Choffnes, E. Katz-Bassett, P. Gill — SIGCOMM 2026

Discussion. In-class exercise: summarize the same speed-test measurements once per test, once per IP address, and once with equal weight per region. What does each estimate represent, how does carrier-grade NAT change the per-address view, and what can any of them say about a region with no observations?

Presenter: Fabián.

Measurements come from whoever happens to be observed. Learn to reason about the unit of analysis (a test, an address, a user, a region), what a sample represents, and how weighting changes the population a number describes.

Thu 1/28 From archived observations to a defensible claim

Review

‘Quis mensurat ipsos mensores?’ Evaluating Data Consistency of BGP Route Collectors & Brokers, P. Mani, Q. Misell, P. Hennen, A. Feldmann, T. Fiebig — IMC 2026

Exercise Revisit a claim from Week 3: what happens if a collector, region, or measurement interval is missing?

Discussion. Bring back Quinzi et al. from Week 3: which checks would you run on the BGP archive before attributing a burst of updates to an RPKI change?

Presenter: Fabián.

Between the vantage point and your analysis sits a collection and archive pipeline. Learn to account for what it retained, not only for what could have been seen, before you make a claim.

Week 5

Tue 2/2 Where is the infrastructure?

Question

The Aleph: Decoding DNS PTR Records With LLMs, K. Thiagarajan et al., CoNEXT 2025

Discussion. Defend one router-location inference from your warm-up using the observations available to you. Which clues are independent? Bring one ambiguous result to class.

Presenter: TBD.

First application of the methods to infrastructure: combining independent, imperfect clues into an inference, and knowing when to narrow, revise, or report unknown.

Thu 2/4 Measuring network infrastructure

Review

Calypso’s Voyage: Charting Traceroute Paths Through Submarine Cables, C. J. Wang, Y. Zhang, Q. Dong, E. Carisimo, R. Durairajan, F. E. Bustamante — IMC 2026

Review

Internet in Africa: A Quantitative Study of the Impact of Infrastructure Development, Y. Wei, J. He, A. Phokeer, T. A. Benson, G. Tyson — IMC 2026

Discussion. Where is the infrastructure, who shares it, and which traffic traverses it? What changes when a cable, facility, or interconnection is added, and what would you need to attribute the change to it?

Presenters: TBD.

Connecting observations to physical infrastructure, and asking what longitudinal data can establish about the effect of a change to that infrastructure.

Week 6

Tue 2/9 DNS: who answers the request?

Review

Routes and Roots: Uncovering the Origins of DNS Root Server Interception, B. Kertmen, D. Tumer, Y. Nosyk, O. Cetin, Q. Lone — CoNEXT 2026

Review

Exploring The Missing Half: A Dual-Perspective Measurement of Encrypted Recursive-to-Authoritative DNS, Y. Zhu, L. Zhang, B. Li, Z. Li et al. — IMC 2026

Alt (second slot): BigTime – Characterizing Large Time Service Providers, P. Huppert, G. Moura, M. Davids, P.-T. de Boer et al., CoNEXT 2026

Discussion. Draw the DNS path and mark where each study observes it. Which comparison would distinguish expected resolver behavior from interception or deployment differences?

Presenters: TBD.

Shared services are hard to observe because many parties sit on the request path. Learn to map where each vantage point sees that path, and what its position rules out.

Thu 2/11 Midterm project presentations Midterm presentations

Short presentations + Q&A.

Each team presents its question, first measurement or pilot result, and the next comparison it needs, while there is still time to fix the design and collect evidence.

Week 7

Tue 2/16 Web measurement and privacy

Review

Crawling in the Deep: Evaluating Web Crawling Configurations for Web Privacy Measurements, K. Ramakrishnan, Q. Xie, U. Anand, F. Li — IMC 2026

Review

Eadem, Sed Aliter: Investigating the Instability of Browser Fingerprinting, X. Lin, S. Wu, J. A. Eckfeldt, P. Sun, J. Polakis — IMC 2026

Discussion. Specify a crawl that could support a claim about observed tracking behavior. If the same browser yields a different fingerprint on a later visit, is the change in the instrument, the browser, or the site, and how would you design repeated measurements to tell?

Presenters: TBD.

Measurement configuration and timing are part of the result. On the web, how you crawl and when you measure determine which behavior you can claim to have observed, and whether someone else could observe it again.

Thu 2/18 Measuring deployed AI services

Review

Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways, G. Lin, Y. Wan, S. Pei, T. Xu, K. Xu, G. Xue — IMC 2026

Review

Can You See Me, GenAI? Characterizing Video-Based Conversational Interaction with Generative AI, R. Cheng, G. Xie, B. Han — IMC 2026 (selected methods and results)

Discussion. What can an external audit establish about the service a customer actually receives? For one claim from either paper, specify a baseline, a controlled variation, and an observable outcome.

Presenters: TBD.

Applying the methods to a new kind of system under measurement: deployed AI services, observed from the outside.

Week 8

Tue 2/23 How do services fail and fail over?

Review

Belt and Suspenders: A Study of Resilience in TikTok’s Global Video Delivery, Y. Zhang, D. Jang, F. E. Bustamante — CoNEXT 2026

Review

Horizon: A Hyper-Edge Observability Engine for Live Streaming Networks, Z. Liu, D. Ding, R. Han, S. Guo et al. — SIGCOMM 2026

Alt: Who Holds the Steering Wheel? Opacity and Consolidation in CDN Replica Selection, R. Kumar, F. E. Bustamante, M. Flores, NINeS 2026

Discussion. What failure does the client-side intervention approximate, and which observations support the inferred failover design (e.g., video-on-demand vs. live)? What would a provider-side view need to observe to confirm or refute it?

Presenters: TBD.

Studying failure means waiting for one, inducing one, or instrumenting the system to catch them. Compare a client-side intervention that approximates a failure with a provider's view of how failures are detected and localized at the edge.

Thu 2/25 How does a LEO network affect an application?

Review

Dissecting the StarLink: Characterizing Queuing and Flow Dynamics in the Starlink Network, H. Cech, N. Mohan, J. Ott — SIGCOMM 2026

Review

Looking at Stars: Analyzing Real-Time Video Conferencing over Starlink, J. Fries, Y. Matezki, R. Bose, N. Mohan — IMC 2026

Discussion. Trace the evidence from network event to transport response to application outcome. Where is the attribution weakest, and what measurement would strengthen it?

Presenters: TBD.

Attribution across layers: linking a network event to its effect on an application, and finding where the chain of evidence is weakest.

Week 9

Tue 3/2 Resilience in the wild: what can an observed outage establish?

Review

Hanging by a Thread: A Study of Single-Upstream Networks, M. Tashiro, R. Fontugne, K. Fukuda, R. Bush, D. Madory — IMC 2026

Review

When the Lights Go Out: Internet Resilience During the 2025 Iberian Power Outage, O. Hohlfeld, Y. Alhamwy, T. Striffler, D. Wagner, M. Wichtlhuber — SIGCOMM CCR 2026 (selected sections)

Discussion. Align the evidence on a timeline, identify which observation supports each attribution, and name a plausible competing explanation.

Presenters: TBD.

Outages are natural experiments. Revisit Week 3's tools on unplanned events and compare them with the induced failures of Week 8.

Thu 3/4 Networks in context: measuring digital sovereignty

Review

EU Digital Sovereignty in Practice: A Technical Assessment of Digital Infrastructure Dependencies, J. Brandl, F. Holzbauer, A. Kaplan, J. Ullrich, A. Dabrowski — IMC 2026

Background

Of Choices and Control – A Comparative Analysis of Government Hosting, R. Kumar et al., IMC 2024 (selected sections on hosting and cross-border dependencies)

Discussion. Classify the same ministry website using both approaches, varying where content is served, who operates the service, and which external services it depends on. Do the approaches agree?

Presenters: TBD.

Networks in context: when a policy question drives the measurement, the definitions you choose determine what you count and which conclusions those counts support.

Week 10

Tue 3/9 Censorship, privacy, and what we can observe

Background

Analysis of Country-wide Internet Outages Caused by Censorship, A. Dainotti et al., IMC 2011 (selected sections)

Review

Discovering Encrypted Client Hello Deployments by Active ECH Probing, J. Mücke, K. Gasser, T. C. Schmidt, M. Wählisch — IMC 2026

Discussion. What evidence would distinguish a service that advertises ECH, one that supports it, and a connection that actually uses it? Which can a passive observer establish, and which require an active experiment?

Presenter: TBD.

Closes the loop on Week 2: what can an observer still infer when a protocol change removes the signals measurements relied on?

Thu 3/11 WINE 2027 PC meeting

Students debate and decide “accepts” — SIGCOMM-style.

Week Finals

Final project presentations (finals week, 3/15–3/20; date TBD)

Reproducibility project talks.

Reading, Reviews & Presentations

Mon, 01 Jan 0001 00:00:00 +0000

Syllabus Calendar & Readings Course Project Reading, Reviews & Presentations

This course is a seminar, which means:

You read the papers before class
You contribute to discussion
You review papers
You present a paper

Here is what that entails.

Reading & Pre-Class Questions

Each week includes several required papers labeled:

Background — foundational context
Question — requires answering a pre-class question
Review — requires deeper reading and critique

Before each class, you must answer a short question on Canvas for every
Question or Review paper for which you are not the assigned presenter or reviewer.

These are graded for comprehension — short but thoughtful answers are enough.

Helpful resources

S. Keshav, How to Read a Paper, CCR 2007
M. Hanson & D. McNamee, Efficient Reading of Papers in Science and Technology

Writing Reviews — WINE 2027 Mini-Conference

The class runs a small program-committee–style process using:

WINE (We do INternet Experimentation) 2027 HotCRP (link TBD)

You will:

Write several conference-style reviews
Evaluate originality, significance, clarity, and methodology
Participate in a PC meeting to discuss and “accept/reject” papers
Lead discussion for at least one paper as a reviewer

Presenting Papers

Every student will present at least one paper.
Presentations are typically 15–20 minutes plus discussion.

A good presentation demonstrates:

Understanding of the problem
Clarity about the methodology
Awareness of limitations
Ability to explain why the paper matters
Critical insight beyond regurgitating the text

Helpful guides

M. Ernst, Advice on Giving Talks
S. Peyton Jones et al., How to Give a Great Research Talk

Slides should be posted to Canvas before class.

Participation Expectations

You’re expected to:

Engage with questions during discussion
Challenge assumptions
Offer alternative explanations
Connect papers across weeks
Bring your own research experiences into conversation

What you get out of a seminar is strongly correlated with what you put in.

Course Project

Mon, 01 Jan 0001 00:00:00 +0000

Syllabus Calendar & Readings Course Project Reading, Reviews & Presentations

The project component includes two hands-on assignments:
a warm-up project to develop essential skills, and a reproducibility project to deepen your understanding through practice.
Below is a concise overview. Full details live on Canvas.

Warm-Up Project

The warm-up project gives you practical experience with Internet-scale measurement techniques, focusing on analyzing geolocating network routers.

Deliverables

Report: 3–5 pages describing your methodology, findings, and analysis.
Presentation: A concise 5-minute presentation summarizing your work.

Reproducibility Project

For the second assignment, you will work in small teams to reproduce results from a published research paper.
This is designed to help you deeply understand a significant networking paper and its methodology.

You will:

Choose a paper (from the course proposed list or your own).
Identify the specific component you intend to reproduce.
Outline your approach and scope.
Contact the original authors (strongly encouraged!).
Meet at least once with course staff for a mid-quarter check-in.

Final Deliverables

Presentation: A short talk summarizing the paper and what you reproduced.
Report:
- 6 pages max, using the HotNets format.
- Appendices/webpage allowed for additional details.

Suggested Report Structure

Abstract – What you did, why it matters, and your top-line results.
Problem Statement – What problem were you addressing?
Prior Work – What has been done, and why wasn’t it enough?
Research Approach – What you built or analyzed; how you approached replication.
Results – Evaluation, insights, unexpected findings.
Lessons Learned + Future Work – What you’d change or extend.
Summary + Conclusion – Wrap-up and implications.

Resources

This project’s structure is inspired by an excellent course originally developed by Nick McKeown at Stanford:
https://reproducingnetworkresearch.wordpress.com/about/

For your paper:

Use the HotNets LaTeX template from the CFP archive
https://conferences.sigcomm.org/hotnets/2024/docs/hotnets24-template.zip
Additional examples and guidance are available on Canvas.