Overview

How would you …

  • Evaluate the effectiveness of a new feature in your startup’s app?
  • Understand the tense dynamics between content providers and ISPs?
  • Analyze the impact of COVID-19 on Internet infrastructure?
  • Assess what drives Internet users’ quality of experience?
  • Distinguish between large-scale failures and deliberate Internet shutdowns?

These seemingly diverse questions share a common thread: the need for experimentation at Internet scale.

Internet-Scale Experimentation is a graduate-level course that blends an introduction to the field with a seminar format, offering a comprehensive exploration of the unique challenges of conducting large-scale experimentation and measurement in networked systems.

Over the past few decades, networked systems have evolved into essential components of daily life and critical foundations of economic, educational, health, and defense systems. While this progress highlights the success and societal impact of our field, it also introduces significant complications for experimentalists. Conducting randomized controlled experiments at even fractions of the scale of today’s systems is nearly impossible. Yet, as we venture into uncharted territories with new ideas, we are increasingly expected to provide compelling evidence of the effects of interventions.

This course equips you with tools and methodologies to meet these challenges. By combining foundational knowledge with hands-on learning, you will:

  • Engage with leading research by reading, reviewing, and presenting academic papers.
  • Participate in discussions on real-world challenges and solutions for Internet-scale experimentation.
  • Conduct your own research project(s), leveraging existing platforms and datasets to investigate critical questions.

This hybrid introduction and seminar structure offers both foundational insights into Internet-scale experimentation and a platform to grapple with its complexities. Through this approach, you’ll develop the skills to design and evaluate impactful experiments in an increasingly networked world.


Topics include

  • Strategies for sound Internet measurements
  • Experimental design and methodologies
  • Measurement tools and platforms
  • Network infrastructure and traffic analysis
  • Applications and distributed services: DNS, Web, Video, Social Networks, …
  • Privacy, security, and policy implications

Course Organization

The course is structured into two main parts:

  1. A five-week introduction to foundational concepts and methods.
  2. A series of in-depth paper discussions and project work.

The project component includes:

  • A warm-up project to build essential skills.
  • A reproducibility project to apply and deepen your understanding by reproducing results from a published paper.

Most class meetings are centered around paper presentations and discussions. You should read each paper before coming to class and be prepared to discuss it.

What you get out of the course depends on what you put into it—this is especially true for seminars. You should read all papers tagged as Background, Question, and Review before coming to class and be prepared to engage. As a reading enforcer, before each class you must answer (in Canvas) a short question for all required readings (i.e., papers tagged as Question or Review) for which you are not the assigned reviewer.

The class will run as a mini-conference with you as Program Committee members. We will use a subset of papers in the schedule (those tagged Review) as our “submissions.” Each of you will write reviews for several of them. We will hold a PC-style meeting at the end of the quarter to decide which papers “should be accepted” for publication.

While there is no required textbook, a classic reference is:

M. Crovella and B. Krishnamurthy, Internet Measurement: Infrastructure, Traffic and Applications, Wiley 2006.


Communication Channels

  • The course web site (this Hugo site + Canvas) will carry announcements, schedule updates, and corrections. Check regularly.
  • Piazza (linked from Canvas) will be used for general discussions and to help identify projects and groups.
  • Canvas and email may be used for class discussion and logistics.
  • For email to course staff, please start your subject line with:
    ISE: <short-topic>
    to ensure a prompt response.

Grading

Grading is criterion-referenced, not curved: your grade depends on how well you do relative to predefined performance levels, not how others perform.

If a test has 100 possible points, for example, then:

  • ≥ 90 → in the A range (90–92: A-, etc.)
  • ≥ 80 → in the B range
  • ≥ 70 → in the C range

If everyone works hard and earns ≥ 90, everyone gets an A.

Total scores (0–100) are determined roughly as:

  • Paper discussion participation (and questions): 10%
  • Paper reviews: 20%
  • PC meeting participation: 5%
  • Paper presentation: 20%
  • Warm-up project: 15%
  • Reproducibility project: 30%