Overview

How would you …

  • Evaluate the effectiveness of a new feature in your startup’s app?
  • Understand the tense dynamics between content providers and ISPs?
  • Analyze the impact of COVID-19 on Internet infrastructure?
  • Assess what drives Internet users’ quality of experience?
  • Distinguish between large-scale failures and deliberate Internet shutdowns?

These seemingly diverse questions share a common thread: the need for experimentation at Internet scale.

Internet-Scale Experimentation is a graduate-level course that blends an introduction to the field with a seminar format, offering a comprehensive exploration of the unique challenges of conducting large-scale experimentation and measurement in networked systems.

Over the past few decades, networked systems have evolved into essential components of daily life and critical foundations of economic, educational, health, and defense systems. While this progress highlights the success and societal impact of our field, it also introduces significant complications for experimentalists. Conducting randomized controlled experiments at even fractions of the scale of today’s systems is nearly impossible. Yet, as we venture into uncharted territories with new ideas, we are increasingly expected to provide compelling evidence of the effects of interventions.

This course equips you with the tools and methodologies to meet these challenges. By combining foundational knowledge with hands-on learning, you will:

  • Engage with leading research by reading, reviewing, and presenting academic papers.
  • Participate in discussions on real-world challenges and solutions for Internet-scale experimentation.
  • Conduct your own research project, leveraging existing platforms and datasets to investigate critical questions.

This hybrid introduction and seminar structure offers you both foundational insights into Internet-scale experimentation and a platform to grapple with its complexities. Through this approach, you’ll develop the skills to design and evaluate impactful experiments in an increasingly networked world.

Topics include:

  • Strategies for sound Internet measurements
  • Experimental design and methodologies
  • Measurement tools and platforms
  • Network infrastructure and traffic analysis
  • Applications and distributed services: DNS, Web, Video, Social Networks, …
  • Privacy, security, policy implications

Course Organization

The course is structured into two main parts: a five-week introduction to foundational concepts, followed by a series of in-depth paper discussions. The project component is hands-on, starting with a warm-up project to build essential skills and culminating in a small reproducibility project to apply and deepen your understanding.

Most class meetings will be centered around paper presentations and discussions. You should read each paper before coming to class and be prepared to discuss it.

What you get out of school depends on what you put on it, that is particularly true of seminars. You should read all Background, Question, and Review papers before coming to class and be prepared to discuss them. As a way of reading enforcer, before each class, you must answer (in Canvas) a short question of all required readings (i.e., papers tagged as Question or Review) you are not assigned as a reviewer.

The class will run as mini-conference with you as the Program Committee member. We will use a subset of the papers included in the schedule as our submissions (those marked as Review). Each of you will write reviews for a handful of them. We will discuss the papers in a PC meeting at the end of the quarter to decide what paper “should be accepted” for publication.

While there is no textbook for the course, a great book on Internet measurement is M. Crovella and B. Krishnamurthy, Internet Measurement: Infrastructure, Traffic and Applications, Wiley 2006.


Communication Channels

  • We will use the course Canvas site to post announcements related to the course. You should check that regularly for schedule changes, clarifications, and corrections to assignments, and other course-related announcements.

  • We will use the Piazza site (see the menu option in Canvas) for general discussions, and help you identify potential projects and groups. There is always email for questions that would be inappropriate to post on the newsgroup/discussion-board. When using email to contact the staff please start your subject line with “ISE: helpful-comment” to ensure a prompt response.


Grading

I use a criterion-referenced method to assign your grade; in other words, your grade will be based on how well you do relative to predetermined performance levels, instead of in comparison with the rest of the class. Thus, if a test has 100 possible points, anyone with a score of 90 or greater will get an A (90-92: A-, etc.), those with scores of 80 or greater will get a B, those with scores of 70 or greater will get a C, and so on. Notice that this means that if everyone works hard and gets >90, everyone gets an A.

Total scores (between 0 and 100) will be determined, roughly, as follows:

  • Paper discussion participation (and questions) 10%
  • Paper review 20%
  • PC meeting participation 5%
  • Paper presentation 20%
  • Warm-up project: 15%
  • Reproducibility project 30%