Experimentation Platforms | Aqualab - Northwestern University

Measurement and Experimentation from the Edge

Sun, 01 Aug 2010 00:00:00 +0000

Project Overview

For a long time, measuring the Internet meant measuring it from the places researchers could get accounts: universities, research networks, and testbeds sitting a short hop from a well-provisioned backbone. Those vantage points are convenient and unrepresentative. The paths that matter to users start behind a home router, on a residential access link, at the far end of the network from anything a testbed can see.

This project took the position that the interesting vantage point is the edge, and that the way to reach it is to ride along with software people already run. Rather than asking users to install a measurement tool, we built measurement into applications they were using anyway — which changed both the scale and the realism of what we could observe, and forced us to take seriously the problem of being a good guest on someone else’s connection.

Two Projects

The thread produced two efforts substantial enough to stand on their own.

NEWS came first. Its premise was that a peer-to-peer application already maintains connections to many peers across many networks, and already knows how those connections are performing — in effect a continuous distributed measurement of the network, generated for free as a side effect of normal use. By looking for correlated performance changes across a peer’s connections, and corroborating across users in the same ISP or country, NEWS detected service-level network events from the edge with no probes and no infrastructure.

Dasu carried the approach into experimentation. It began as an ISP characterization tool — measuring the broadband service a user actually received, from inside the home, across many providers and countries — and became a platform that could host other researchers’ experiments from tens of thousands of real access links, with the scheduling and safeguards that hosting implies when the machine belongs to a volunteer.

Both are documented on their own pages; what follows is the tooling that supported them.

Software

The Nixes Tool Set — bash scripts to install, maintain, control and monitor applications on PlanetLab, used to deploy the wide-area experiments behind this work.

The measurement clients themselves — the Vuze plugins and the Dasu standalone client — are described on the NEWS and Dasu pages.

Papers

Crowdsourcing Service-Level Network Event Detection — SIGCOMM 2010
Crowdsourcing ISP Characterization to The Network Edge — W-MUST 2011
Dasu: ISP Characterization from the Edge, A BitTorrent Implementation — SIGCOMM 2011 (demo)
Trying Broadband Characterization at Home — PAM 2013
Dasu: Pushing Experiments to the Internet’s Edge — NSDI 2013
Experiments at the Internet’s Edge with Dasu — NSDI 2013 (demo)
Characterizing Broadband Services with Dasu — NSDI 2013 (demo)
A Measurement Experimentation Platform at the Internet’s Edge — IEEE/ACM Transactions on Networking, 2015

Coverage

NEWS was covered by RedOrbit, Slashdot, and The Hindu in 2008.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Balachander Krishnamurthy (AT&T Labs–Research)
Walter Willinger (AT&T Labs–Research)

AquaLab Research Group

Dasu

Sun, 30 May 2010 00:00:00 +0000

The Short Story

Dasu was a measurement experimentation platform for the Internet’s edge. It supported both controlled network experimentation and broadband characterization — building on public interest in the latter to gain the adoption necessary for the former.

Do you ever wonder what your ISP service is really like? Are you getting your money’s worth? How do public DNS services perform compared with your own ISP’s, over the long run?

Some ISPs set complex policies based on time of day, service level or application; others oversubscribe their networks, shape traffic, or discourage heavy users. To the person paying the bill, those policies show up as time-of-day variation in available bandwidth, poor performance for particular applications, or a general sense that the service is worse than advertised. Dasu’s goal was to capture those variations by continuously monitoring the level of service actually delivered, so users could make informed decisions about their provider.

Why Continuous, and Why From the Edge

Most earlier work on ISP performance relied on one-time active measurements from user-triggered benchmarking tools, which capture only snapshots. That approach cannot see service variation or sporadic events, precisely because such things are transient. Many tools also looked at a single dimension — upload and download caps, DNS performance, HTTP caching.

Done properly, ISP characterization needs to be:

at scale, to cover the diversity of providers and service tiers
continuous, to catch traffic shaping, oversubscription and unscheduled outages
from end users, so what is characterized is the service people actually receive

Existing approaches traded these against each other. Our argument was that network-intensive applications already running on end systems — BitTorrent being the obvious one — avoid the trade-off entirely.

By passively monitoring user-generated traffic, Dasu captured the end user’s view at scale. By combining that with on-demand active measurements, it reached the effectiveness of hardware-based approaches such as SamKnows and the FCC’s programme, without their cost, while keeping the flexibility and low adoption barrier of software.

Distribution

Dasu shipped first as a plugin for the Vuze/Azureus BitTorrent client, then as a standalone client for users who did not run BitTorrent — a small application that sat in the status bar monitoring ISP performance, with installers for Windows and macOS 10.5+. Versions ran from 1.0 in May 2010 through 5.0 in 2013, with automatic updates.

It was released under the GPL, with source included in the distributed jar. Neither the plugin nor the standalone client is available today.

Why the Name

Dasu is the Japanese word for putting something out there, or showing something — used especially for showing secrets.

Release history (2010–2013)

24 Apr 2013 — version 5.0 to beta testers
15 Jan 2013 — version 4.11 released; prevented loss of window focus on Windows during test execution
25 Oct 2012 — version 4.0 released
9 Apr 2011 — version 3.0 released, a major rewrite
5 Nov 2010 — fix for a client incompatibility
3 Nov 2010 — version 2.0, detecting more types of interference
30 May 2010 — first public release, version 1.0

Papers

Crowdsourcing ISP Characterization to The Network Edge — W-MUST 2011
Dasu: ISP Characterization from the Edge, A BitTorrent Implementation — SIGCOMM 2011 (demo)
Up, Down and Around the Stack: ISP Characterization from Network Intensive Applications — W-MUST 2012, also in CCR
Trying Broadband Characterization at Home — PAM 2013
Dasu: Pushing Experiments to the Internet’s Edge — NSDI 2013
Experiments at the Internet’s Edge with Dasu — NSDI 2013 (demo)
Characterizing Broadband Services with Dasu — NSDI 2013 (demo)
A Measurement Experimentation Platform at the Internet’s Edge — IEEE/ACM Transactions on Networking, 2015

Coverage

Ars Technica covered our analysis of BitTorrent usage during the 2011 Japanese earthquake and the Internet shutdowns in Egypt and Libya.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Balachander Krishnamurthy (AT&T Labs–Research)
Walter Willinger (AT&T Labs–Research)

Measurement and Experimentation from the Edge — the wider thread this belongs to
NEWS — the earlier system that established the approach

The Nixes Tool Set

Sat, 01 Jan 2005 00:00:00 +0000

Overview

Nixes provided a set of bash scripts to install, maintain, control and monitor applications on PlanetLab. It bootstrapped nodes with yum and let the user install RPMs from the PlanetLab distribution. It was designed for simplicity and performance, and had three parts: the scripts, a configuration file, and a repository — a public web directory — hosting the application being deployed.

This is the kind of infrastructure that rarely gets written up but without which none of the wide-area experiments happen.

The Tools

Every tool took a list of nodes — fully qualified Internet addresses — and authenticated to them over SSH with a public/private key pair named in the configuration file.

plsetup — bootstraps the vserver with yum, gzip and Java
plinstallrpm — installs RPMs across all nodes, resolving dependencies automatically
pldeploy — deploys a file structure to the nodes
plcmd — executes an arbitrary set of commands on all nodes, named by an environment variable holding the script

All of them ran in parallel, thirty threads by default, and wrote a per-node log into the directory named by TMP. Only success was reported to the console; a failure sent you to the logs. Running several tools in parallel yourself was explicitly discouraged — the logs collide.

Configuration

A .nixesrc in the user’s home directory configured everything. Options included the SSH identity, the PlanetLab slice name, the location of the bootstrap script, and the base location and list of files and executables to deploy. Options could equally be set in the environment, sharing a namespace with the file.

Bootstrapping

Bootstrapping downloaded a script to each node and ran it, installing yum so that any RPM could subsequently be pulled by name. Custom RPMs not on the Red Hat mirrors could be added at the top of bootstrap.sh.

Java was a special case. Because of Sun’s licensing at the time, users had to download the JDK themselves and host it on a server only they knew about, then update the location in the bootstrap script — a detail that says a good deal about the era.

Implementation and Availability

The scripts were written in bash, with static interleaved scheduling; dynamic scheduling was noted as a possible improvement that was never made. Nixes was distributed to users who registered on the lab site, and is no longer available.

Feedback and bug reports went to Stefan Birrer.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Stefan Birrer

Measurement and Experimentation from the Edge
Resilient Overlays and Multicast — Nixes supported those wide-area experiments

Experimentation Platforms | Aqualab - Northwestern University

Measurement and Experimentation from the Edge

Project Overview

Two Projects

Software

Papers

Coverage

People

Related Links

Dasu

The Short Story

Why Continuous, and Why From the Edge

Distribution

Why the Name

Papers

Coverage

People

Related Links

The Nixes Tool Set

Overview

The Tools

Configuration

Bootstrapping

Implementation and Availability

People

Related Links