CS 445 Internet-Scale Experimentation — Winter 2025

News and InformationOverviewCalendarReading, Writing, PresentingProjects

ANNOUNCEMENTS

Remember to check this (and Canvas) regularly!

ADMINISTRATIVE INFORMATION

STAFF

Professor

Fabián E. Bustamante
Seely Mudd #3905
+1 847 491-2745
fabianb@cs…

TA

Kedar Thiagarajan
Seely Mudd
kedarthiagarajan2028@…

LOCATION AND TIME

Lectures: TuTh 11:00-12:20AM | Tech Institute A110
Professor Office Hours: TBD

CATALOG DESCRIPTION

Internet-scale Experimentation is a graduate-level course that combines an introduction to the field with a seminar component, delving into the challenges of conducting experimentation and measurements in large-scale networked systems.
This course satisfies the Systems requirement and the project requirement.

COURSE PREREQUISITES

CS 340 “Intro to Computer Networking” or CS 345 “Distributed Systems”.

If you have taken similar courses somewhere else or have not taken any of these courses, but would like to register for this seminar, please contact me.

Disability

In compliance with Section 504 of the 1973 Rehabilitation Act and the Americans with Disabilities Act, Northwestern University is committed to providing equal access to all programming. Students with disabilities seeking accommodations are encouraged to contact the office of Services for Students with Disabilities (SSD) at +1 847 467-5530 or ssd@northwestern.edu. SSD is located in the basement of Scott Hall. Additionally, I am available to discuss disability-related needs during office hours or by appointment.

How would you …

  1. Evaluate the effectiveness of a new feature in your startup’s app?
  2. Understand the tense dynamics between content providers and ISPs?
  3. Analyze the impact of COVID-19 on Internet infrastructure?
  4. Assess what drives Internet users’ quality of experience?
  5. Distinguish between large-scale failures and deliberate Internet shutdowns?

These seemingly diverse questions share a common thread: the need for experimentation at Internet scale.

Internet-Scale Experimentation is a graduate-level course that blends an introduction to the field with a seminar format, offering a comprehensive exploration of the unique challenges of conducting large-scale experimentation and measurement in networked systems.

Over the past few decades, networked systems have evolved into essential components of daily life and critical foundations of economic, educational, health, and defense systems. While this progress highlights the success and societal impact of our field, it also introduces significant complications for experimentalists. Conducting randomized controlled experiments at even fractions of the scale of today’s systems is nearly impossible. Yet, as we venture into uncharted territories with new ideas, we are increasingly expected to provide compelling evidence of the effects of interventions.

This course equips you with the tools and methodologies to meet these challenges. By combining foundational knowledge with hands-on learning, you will:

  • Engage with leading research by reading, reviewing, and presenting academic papers.
  • Participate in discussions on real-world challenges and solutions for Internet-scale experimentation.
  • Conduct your own research project, leveraging existing platforms and datasets to investigate critical questions.

This hybrid introduction and seminar structure offers you both foundational insights into Internet-scale experimentation and a platform to grapple with its complexities. Through this approach, you’ll develop the skills to design and evaluate impactful experiments in an increasingly networked world.

Topics include:

  • Strategies for sound Internet measurements
  • Experimental design and methodologies
  • Measurement tools and platforms
  • Network infrastructure and traffic analysis
  • Applications and distributed services: DNS, Web, Video, Social Networks, …
  • Privacy, security, policy implications

COURSE ORGANIZATION

The course is structured into two main parts: a five-week introduction to foundational concepts, followed by a series of in-depth paper discussions. The project component is hands-on, starting with a warm-up project to build essential skills and culminating in a small reproducibility project to apply and deepen your understanding.

Most class meetings will be centered around paper presentations and discussions. You should read each paper before coming to class and be prepared to discuss it.

What you get out of school depends on what you put on it, that is particularly true of seminars. You should read all Background, Question and Review papers before coming to class (see tags in the Calendar tab) and be prepared to discuss them. As a way of reading enforcer, before each class, you must answer (in Canvas) a short question of all required readings (i.e., papers tagged as Question or Review papers) you are not assigned as a reviewer.

The class will run as mini-conference with you as the Program Committee member. We will use a subset of the papers included in the schedule as our submissions (those marked as Review). Each of you will write reviews for a handful of them. We will discuss the papers in a PC meeting at the end of the quarter to decide what paper “should be accepted” for publication.

While there is no textbook for the course, a great book on Internet measurement is
M. Crovella and B. Krishnamurthy, Internet Measurement: Infrastructure, Traffic and Applications, Wiley 2006.

COMMUNICATION CHANNELS

There are a number of communication channels set up for this class:

  • We will use the course web site (here and Canvas) to post announcements related to the course. You should check this regularly for schedule changes, clarifications and corrections to assignments, and other course-related announcements.
  • We will use the Piazza site (see the menu option in Canvas) for general discussions, and help you identify potential projects and groups.
  •  
  • We will use Canvas or email for class discussion.
  • There is always email for questions that would be inappropriate to post on the newsgroup/discussion-board. When using email to contact the staff please start your subject line with “ISE: helpful-comment” to ensure a prompt response.

GRADING

I use a criterion-referenced method to assign your grade; in other words, your grade will be based on how well you do relative to predetermined performance levels, instead of in comparison with the rest of the class. Thus, if a test has 100 possible points, anyone with a score of 90 or greater will get an A (90-92: A-, etc.), those with scores of 80 or greater will get a B, those with scores of 70 or greater will get a C, and so on. Notice that this means that if everyone works hard and gets >90, everyone gets an A.

Total scores (between 0 and 100) will be determined, roughly, as follows:

  • Paper discussion participation (and questions) 10%
  • Paper review 20%
  • PC meeting participation 5%
  • Paper presentation 20%
  • Warm-up project: 15%
  • Reproducibility project 30%

(The list of papers may change slightly before the beginning of the term.)

Week Date Topics and Reading Due
1
01/07

Course introduction

Before class, read How to read a paper, S. Keshav SIGCOMM CCR 2007.
01/09

Internet Architecture – A Brief Overview

2
01/14

Internet-Scale Experimentation – The Questions

01/16

Measurement tools and platforms

3
01/21

Measurement tools and platforms

01/23

Measurement approaches and datasets

4
01/28

Experimental approaches

01/30

Experimental approaches

Warm-up project
5
02/04

Weighting your measurements

Reproducibility project proposal
02/06

Weighting your measurements

6
02/11

Infrastructure

02/13

Infrastructure

7
02/18

Resilience

02/20

DNS

Reproducibility project check-in
8
02/25

Web

02/27

Video

9
03/04

Networks in context

03/06

Privacy

10
03/11

Policy

Reproducibility project report
03/13

PC Meeting

FINALS 03/17

12-2PM Finals presentations (place TBD)

READING, WRITING AND PRESENTING PAPERS

READING AND ANSWERING QUESTIONS

We will be reading two or more papers per week. The papers will be first presented to the group by one or more students and then discussed in a round-table manner.

To ensure lively discussions, you will be responsible for reading the assigned papers before each class. I will post a question about each paper 24hr before class. Your answer need only be long enough to demonstrate that you understand the paper; a paragraph or two should be enough. I will check your answers to make sure they make sense and they will count for part of the paper discussion grade.

You may find the following documents useful:

WRITING REVIEWS

At one time or another, every researcher is asked to review papers submitted for publication at a conference or journal; a process known as peer review. We will work on this skill by running a mini-conference – WINE (We do INternet Experimentation) 2025.

All class members will be part of the “Programm Committee” for our mini-conference and we will consider all papers listed in our schedule as our submissions.

Each paper will receive three reviews. We will discuss all papers in a two-part PC meeting (around midterm and during final weeks) to decide what paper “should be accepted” for publication. Each paper discussion will be led by one of the reviewers (assigned by the PC chair).

You may find the following documents useful:

To enter your reviews go to WINE 2025 (Access is restricted to Northwestern).

PRESENTING

Most class meetings will be centered around a paper presentation and discussion. Each student will be responsible for presenting one of the papers in the schedule (so, if you haven’t yet, please email me three ranked options).

Giving a good presentation is hard work. Please make sure to allocate enough time to prepare for yours. There are some good pointers around that you may want to look at.

Take a look at this excellent presentation by Peyton Jones How to give a great research talk
S. Peyton Jones, J. Launchbury, J. Hughes, SIGPLAN Notices 28(11), Nov 1993.

PROJECTS

The project component includes two hands-on assignments: a warm-up project to develop essential skills and a small reproducibility project to deepen your understanding and experience. Below is a brief overview of each. Full details can be found on the course Canvas site.

WARM-UP PROJECT

The warm-up project provides practical experience with internet-scale measurement techniques, focusing on analyzing cross-border dependencies in government hosting. This project has two deliverables:

  • Report: A 3-5 page document detailing your methodology, findings, and analysis.
  • Presentation: A concise 5-minute presentation summarizing your work.

REPRODUCTION PROJECT

For the second assignment, you will work in small teams to reproduce results from a published paper. This project is designed to give you an in-depth understanding of a significant research paper, its underlying concepts, and its key findings. It emphasizes the rewarding aspects of research—building a prototype or experimental framework and conducting experiments—while addressing an established, meaningful problem.

Steps and Expectations

  • Paper Selection: Choose a paper from the course reading list or propose one of your own.
  • Project Planning: Identify the specific part of the work you intend to reproduce and outline your approach.
  • Engage with Authors: Reach out to the paper’s original authors to inform them of your project and seek advice. Course staff are available to support this step.
  • Check-Ins: Meet with course staff at least once during the second half of the term to discuss your progress.

Final Deliverables

  1. Presentation: A short class presentation highlighting the key insights from the paper and your reproducibility efforts.
  2. Report: A final report following the HotNets workshop format, limited to 6 pages. Appendices or a webpage may be used for additional details.

Suggested Report Structure

  • Abstract: What did you do? Why is it important? What are your high-level results?
  • Problem Statement: What problem were you addressing?
  • Prior Work: How have others approached this problem, and why were those efforts insufficient?
  • Research Approach: What methodology did you use? What did you design or build?
  • Results: What were your findings? How did you evaluate your work?
  • Lessons Learned and Future Work: What would you do differently, knowing what you know now? What questions remain?
  • Summary and Conclusions: Summarize your work and its implications.

RESOURCES

The project draws inspiration from an excellent course originally developed by Nick McKeown at Stanford University. Examples of past projects from the Stanford course are available to guide your work.
For your report, a LaTeX template and example paper are provided in the HotNets CFP archive and on the course Canvas site.