Peer-to-Peer | Aqualab - Northwestern University

Measurement and Experimentation from the Edge

Sun, 01 Aug 2010 00:00:00 +0000

Project Overview

For a long time, measuring the Internet meant measuring it from the places researchers could get accounts: universities, research networks, and testbeds sitting a short hop from a well-provisioned backbone. Those vantage points are convenient and unrepresentative. The paths that matter to users start behind a home router, on a residential access link, at the far end of the network from anything a testbed can see.

This project took the position that the interesting vantage point is the edge, and that the way to reach it is to ride along with software people already run. Rather than asking users to install a measurement tool, we built measurement into applications they were using anyway — which changed both the scale and the realism of what we could observe, and forced us to take seriously the problem of being a good guest on someone else’s connection.

Two Projects

The thread produced two efforts substantial enough to stand on their own.

NEWS came first. Its premise was that a peer-to-peer application already maintains connections to many peers across many networks, and already knows how those connections are performing — in effect a continuous distributed measurement of the network, generated for free as a side effect of normal use. By looking for correlated performance changes across a peer’s connections, and corroborating across users in the same ISP or country, NEWS detected service-level network events from the edge with no probes and no infrastructure.

Dasu carried the approach into experimentation. It began as an ISP characterization tool — measuring the broadband service a user actually received, from inside the home, across many providers and countries — and became a platform that could host other researchers’ experiments from tens of thousands of real access links, with the scheduling and safeguards that hosting implies when the machine belongs to a volunteer.

Both are documented on their own pages; what follows is the tooling that supported them.

Software

The Nixes Tool Set — bash scripts to install, maintain, control and monitor applications on PlanetLab, used to deploy the wide-area experiments behind this work.

The measurement clients themselves — the Vuze plugins and the Dasu standalone client — are described on the NEWS and Dasu pages.

Papers

Crowdsourcing Service-Level Network Event Detection — SIGCOMM 2010
Crowdsourcing ISP Characterization to The Network Edge — W-MUST 2011
Dasu: ISP Characterization from the Edge, A BitTorrent Implementation — SIGCOMM 2011 (demo)
Trying Broadband Characterization at Home — PAM 2013
Dasu: Pushing Experiments to the Internet’s Edge — NSDI 2013
Experiments at the Internet’s Edge with Dasu — NSDI 2013 (demo)
Characterizing Broadband Services with Dasu — NSDI 2013 (demo)
A Measurement Experimentation Platform at the Internet’s Edge — IEEE/ACM Transactions on Networking, 2015

Coverage

NEWS was covered by RedOrbit, Slashdot, and The Hindu in 2008.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Balachander Krishnamurthy (AT&T Labs–Research)
Walter Willinger (AT&T Labs–Research)

AquaLab Research Group

Dasu

Sun, 30 May 2010 00:00:00 +0000

The Short Story

Dasu was a measurement experimentation platform for the Internet’s edge. It supported both controlled network experimentation and broadband characterization — building on public interest in the latter to gain the adoption necessary for the former.

Do you ever wonder what your ISP service is really like? Are you getting your money’s worth? How do public DNS services perform compared with your own ISP’s, over the long run?

Some ISPs set complex policies based on time of day, service level or application; others oversubscribe their networks, shape traffic, or discourage heavy users. To the person paying the bill, those policies show up as time-of-day variation in available bandwidth, poor performance for particular applications, or a general sense that the service is worse than advertised. Dasu’s goal was to capture those variations by continuously monitoring the level of service actually delivered, so users could make informed decisions about their provider.

Why Continuous, and Why From the Edge

Most earlier work on ISP performance relied on one-time active measurements from user-triggered benchmarking tools, which capture only snapshots. That approach cannot see service variation or sporadic events, precisely because such things are transient. Many tools also looked at a single dimension — upload and download caps, DNS performance, HTTP caching.

Done properly, ISP characterization needs to be:

at scale, to cover the diversity of providers and service tiers
continuous, to catch traffic shaping, oversubscription and unscheduled outages
from end users, so what is characterized is the service people actually receive

Existing approaches traded these against each other. Our argument was that network-intensive applications already running on end systems — BitTorrent being the obvious one — avoid the trade-off entirely.

By passively monitoring user-generated traffic, Dasu captured the end user’s view at scale. By combining that with on-demand active measurements, it reached the effectiveness of hardware-based approaches such as SamKnows and the FCC’s programme, without their cost, while keeping the flexibility and low adoption barrier of software.

Distribution

Dasu shipped first as a plugin for the Vuze/Azureus BitTorrent client, then as a standalone client for users who did not run BitTorrent — a small application that sat in the status bar monitoring ISP performance, with installers for Windows and macOS 10.5+. Versions ran from 1.0 in May 2010 through 5.0 in 2013, with automatic updates.

It was released under the GPL, with source included in the distributed jar. Neither the plugin nor the standalone client is available today.

Why the Name

Dasu is the Japanese word for putting something out there, or showing something — used especially for showing secrets.

Release history (2010–2013)

24 Apr 2013 — version 5.0 to beta testers
15 Jan 2013 — version 4.11 released; prevented loss of window focus on Windows during test execution
25 Oct 2012 — version 4.0 released
9 Apr 2011 — version 3.0 released, a major rewrite
5 Nov 2010 — fix for a client incompatibility
3 Nov 2010 — version 2.0, detecting more types of interference
30 May 2010 — first public release, version 1.0

Papers

Crowdsourcing ISP Characterization to The Network Edge — W-MUST 2011
Dasu: ISP Characterization from the Edge, A BitTorrent Implementation — SIGCOMM 2011 (demo)
Up, Down and Around the Stack: ISP Characterization from Network Intensive Applications — W-MUST 2012, also in CCR
Trying Broadband Characterization at Home — PAM 2013
Dasu: Pushing Experiments to the Internet’s Edge — NSDI 2013
Experiments at the Internet’s Edge with Dasu — NSDI 2013 (demo)
Characterizing Broadband Services with Dasu — NSDI 2013 (demo)
A Measurement Experimentation Platform at the Internet’s Edge — IEEE/ACM Transactions on Networking, 2015

Coverage

Ars Technica covered our analysis of BitTorrent usage during the 2011 Japanese earthquake and the Internet shutdowns in Egypt and Libya.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Balachander Krishnamurthy (AT&T Labs–Research)
Walter Willinger (AT&T Labs–Research)

Measurement and Experimentation from the Edge — the wider thread this belongs to
NEWS — the earlier system that established the approach

Privacy in Peer-to-Peer Systems

Mon, 01 Jun 2009 00:00:00 +0000

Project Overview

Discussions of privacy in peer-to-peer systems tended to focus on content: what a user downloads, and who can observe it. This project started from a different place. In BitTorrent, whom you connect to is itself revealing, because people with similar interests end up in the same swarms repeatedly, and those repeated co-occurrences are visible to anyone watching the network.

Working with Luís Amaral’s group at Northwestern, we applied community-detection methods from complex-network analysis to BitTorrent connection patterns. The result, reported in Strange Bedfellows, is that users cluster into stable communities that persist over time and can be recovered from connection structure alone — no payload inspection required. An observer who cannot see what you downloaded can still infer a great deal about what you are interested in, simply from the company you keep.

The interdisciplinary collaboration was essential here: the measurement side supplied the traces and the systems questions, and the network-science side supplied the methods for finding structure in them.

SwarmScreen

If connection patterns are the leak, then the defence has to operate on connection patterns. SwarmScreen adds a controlled fraction of connections to swarms the user has no interest in, deliberately blurring the community structure an observer would otherwise recover.

The design point is plausible deniability rather than concealment. A user’s real activity is still there, but it is no longer separable from behaviour they never chose, so an observer cannot attribute any particular interest with confidence. The cost is a tunable amount of extra traffic, which the tech report examines against the privacy gained.

Software

SwarmScreen — a Vuze extension making downloading behaviour difficult to classify from connection patterns, with a tunable privacy/performance trade-off.

Papers

Strange Bedfellows: Communities in BitTorrent — IPTPS 2010
SwarmScreen: Privacy Through Plausible Deniability in P2P Systems — Northwestern EECS technical report, 2009

Coverage

The Register covered the work in April 2009, under “P2P eavesdrop ‘guilt by association attack’ developed”.

People

This was joint work with the Amaral Lab at Northwestern.

Faculty

Fabián E. Bustamante (Northwestern University)
Luís A. Nunes Amaral (Northwestern University)

Students

David R. Choffnes

Collaborators

Jordi Duch (Northwestern University)
Dean Malmgren (Northwestern University)
Roger Guimerà (Northwestern University)

AquaLab Research Group

SwarmScreen

Mon, 09 Mar 2009 00:00:00 +0000

The Short Story

The goal was to make it harder for an attacker to work out your downloading habits. Part of why BitTorrent works so well is that it lets you download from large numbers of connections — and those same connections are opportunities for eavesdropping. Our study of the BitTorrent network showed that user connection patterns reveal strong communities, which enable a guilt-by-association attack: an entire community can be classified by monitoring one of its members. With P2P networks increasingly under surveillance from private and government organizations, this needed a practical answer.

SwarmScreen hid real traffic in a sea of connections to randomly selected torrents, shaping those extra connections to look like the genuine ones. It slowed downloads, necessarily — so it offered a tuning knob, SPF, the SwarmScreen Protection Factor. Like sunscreen, a higher setting gives more protection; a lower one gives back bandwidth.

The Attack

Peer-to-peer systems draw their scalability and performance from large numbers of connections between cooperating hosts. Existing privacy work concealed connection data through encryption and trusted networks, but left the existence of each connection visible.

BitTorrent is a useful case because peers connect purely on shared, concurrent interest in the same content — not friendship, language or geography. Using connection patterns gathered from real users, we studied whether communities form: collections of peers far more likely to connect to each other than to random peers. They do, and strongly. Users inside a typical community were 5 to 25 times more likely to connect to each other than to users outside it.

That structure is what makes the attack work. From a single observation point, an attacker could reveal 50% of the network using only knowledge of a peer’s neighbours and their neighbours — two hops. An attacker monitoring just 1% of the network could correctly assign users to their communities of interest more than 86% of the time.

The Defence

We proposed a privacy-preserving layer that obfuscates user-generated network behaviour, and showed that plausible deniability is achievable by adding a relatively small proportion — between 25% and 50% — of extra random connections, provided they are statistically indistinguishable from natural ones. SwarmScreen generated exactly those, by participating in randomly selected torrents without looking anomalous.

Encryption does not solve this problem, because the attack reads connection patterns rather than payloads. Tor disguises endpoints but is not built for P2P, and downloads through it slowed by roughly a factor of ten; SwarmScreen let users choose their own slowdown instead.

Why the Name

SwarmScreen uses multiple swarms to screen real traffic. It also sounds like sunscreen, which made SPF the natural name for the privacy/performance dial.

Availability

SwarmScreen installed into the Vuze/Azureus client, first released in March 2009, with community translations into French, Italian, Portuguese, Slovak, Russian, Polish, Chinese and Catalan. It is no longer available.

The project shipped with a legal disclaimer worth preserving in spirit: the privacy claims rested on the results in the technical report and were not legal advice, and the software downloaded nothing unless the user configured it to.

Papers

Strange Bedfellows: Communities in BitTorrent — IPTPS 2010
SwarmScreen: Privacy Through Plausible Deniability in P2P Systems — Northwestern EECS technical report, 2009

Coverage

The Register covered the work in April 2009, under “P2P eavesdrop ‘guilt by association attack’ developed”.

People

This was joint work with the Amaral Lab at Northwestern.

Faculty

Fabián E. Bustamante (Northwestern University)
Luís A. Nunes Amaral (Northwestern University)

Students

David R. Choffnes

Collaborators

Jordi Duch, Dean Malmgren, Roger Guimerà (Northwestern University)

Privacy in Peer-to-Peer Systems — the project SwarmScreen was built for

NEWS

Wed, 07 May 2008 00:00:00 +0000

The Short Story

The goal was to reliably find problems in the network and raise alerts about them. As a user, you want to know you are getting the service you pay for, and to be told quickly when something breaks — particularly when an interruption might entitle you to compensation. For an ISP, the same information localizes and identifies faults faster, so they can be fixed before they generate complaints.

NEWS did this by passively monitoring BitTorrent performance and watching for changes that suggested network trouble. Because a problem can be anywhere, including inside your own home network, NEWS corroborated across multiple users in the same area — the same ISP, or the same country. When enough people saw the same problem in the same place, it raised an alarm.

The approach’s virtue was its cost: monitoring traffic the client was generating anyway made detection extremely lightweight.

NEWS coverage. Detection depended on having enough users inside the same ISP or country to corroborate a suspected problem.

Why the Edge

The Internet increasingly carried services — VoIP, content distribution, IPTV — whose user experience had become a benchmark for providers and users alike. That experience is largely determined by the frequency, duration and severity of network events, so there is a clear need to detect, isolate and explain those events quickly.

Most prior work monitored core networks or probed from research and education networks. That is effective for faults affecting many customers at once, but it misses silent failures — incompatible QoS or ACL settings, for instance — and their effect on individual services. End-to-end monitoring approaches, meanwhile, required active measurement that does not scale to the number of elements at the network’s edge.

NEWS took the other route: push performance monitoring and detection onto the end systems themselves. Crowdsourcing the monitoring meant participating hosts could collectively handle the data volume needed for real-time detection at the scale of millions of monitors, and installing inside or alongside a service allowed immediate, incremental deployment.

Distribution

NEWS installed into the Vuze/Azureus BitTorrent client and reached hundreds of thousands of users. It was released under the GPL, with source included in the jar. Users contributed translations. It is no longer available.

Release history (2008)

24 Oct 2008 — a Java-version problem affecting macOS identified, fix planned for the next release
1 Jul 2008 — version 0.8 mainlined, with numerous bug fixes
24 Jun 2008 — version 0.6 released as a signed plugin for beta testing at scale
19 May 2008 — version 0.2, mature enough for general use
7 May 2008 — first version released

Papers

Crowdsourcing Service-Level Network Event Detection — SIGCOMM 2010
Using the Crowd to Monitor the Cloud: Detecting Network Events from Edge Systems — Northwestern EECS technical report, 2009

Coverage

NEWS was covered by RedOrbit, Slashdot, and The Hindu in 2008.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

David R. Choffnes

Measurement and Experimentation from the Edge — the wider thread this belongs to
Dasu — the platform that grew out of the same approach

Ono

Sun, 01 Jul 2007 00:00:00 +0000

Motivation and Approach

The goal was simple: improve download speeds for your BitTorrent client. In most peer-to-peer applications the choice of which peer to download from is essentially arbitrary. When most peers offer good performance the random choice works fine — but if the peers you draw are on the other side of the world, your downloads suffer.

Ono avoided that by proactively finding peers close to you in a networking sense, which generally means better response time and better transfer rates. It identified those peers by reusing the network measurements content delivery networks already perform, rather than probing paths itself. A CDN constantly decides which replica should serve which client; two clients steered to the same replica are, in all likelihood, near each other. Ono read that signal instead of generating its own — needing only infrequent DNS lookups, never active path measurement.

Results

Ono’s coverage map: users in nearly every country, shaded by concentration.

From results collected across participating users — well over 100,000 of them — Ono located peers along paths with two orders of magnitude lower latency and 30% lower loss rates than those BitTorrent picked at random.

Those higher-quality paths translated into throughput:

In constrained settings, where peers are limited by available bandwidth, Ono delivered a 31% average improvement in download rate.
Where bandwidth was plentiful, it increased download rates by 207% on average, and improved median rates by 883%.

The result mattered because it contradicted the prevailing assumption of the period. Reducing cross-ISP traffic was understood to be something ISPs wanted and users paid for. Ono showed the two interests could be served at once.

Why the Name

Ono is the Hawaiian word for the King Mackerel, and also means “good to eat” — we hoped users would find the plugin as tasty as the fish, which is likewise known as the wahoo, a sound we hoped they would make on seeing their download speeds. The name also ties the lab’s aquatic theme to Akamai, the CDN whose measurements Ono relied on, and whose name is Hawaiian for “clever”.

Why Not the Alternatives

Vuze already shipped a network coordinate system. As Ledlie et al. had shown, those coordinates were badly inaccurate, and our own independent measurements agreed: only 10% of them carried less than 10% error, while more than 60% were off by 100% or more.

Coarse heuristics had a different problem. Class C subnets and AS numbers scale well, but say little: there may be nobody downloading your file on your subnet, and your AS may span an entire country. Neither reflects current network conditions. Reusing a CDN’s live view avoided both failure modes.

Vuze itself was chosen because it was the most used client of its day, was Java-based so it ran anywhere, and supported plugins — so Ono required no changes to a user’s existing installation and could update itself. Ono carried a 100% Java DNS implementation, and ran wherever Vuze did: tested extensively on Windows and Linux, more lightly on macOS.

Beyond the plugin, we implemented a tracker that used Ono to return nearby peers, and a reduced version of the Ono service in Java for integration elsewhere.

Availability

Ono shipped through the official Vuze plugin channel and reached version 1.8.8 by late 2008, with translations contributed by users in several languages. It was released under the GPL, with source included in the distributed jar.

It is no longer available. Both download hosts of that era (aqua-lab.org and merlot.cs.northwestern.edu) stopped resolving long ago, and the Vuze plugin ecosystem it depended on is gone.

Historical release notes and support issues (2007–2008)

Kept as a record of the deployment.

17 Oct 2008 — GUI updated in Ono 1.8.8, with an updated German translation.
16 Oct 2008 — Vuze 4.0 broke the plugin’s GUI. Ono continued to bias peer selection correctly; only the display of its information was affected.
4 Aug 2008 — Added a map of Ono coverage.
12 May 2008 — Added a link to the paper demonstrating Ono’s effectiveness.
1 May 2008 — Site updated with deployment performance results and developer resources.
9 Jan 2008 — Backend revamped for efficiency; GUI work underway; several translations added.
18 Aug 2007 — Latest versions published through the Vuze plugin list; call for translators.
12 Jul 2007 — Version 1.5.3 released with bug fixes.

Two support issues came up often enough to document:

Vuze took longer to shut down after installing Ono. Expected — Ono was reporting performance statistics for analysis on close. The wait was configurable, and statistics collection could be disabled entirely.
Screen flicker and raised CPU on Windows Vista. A rare interaction, resolved by reducing the number of simultaneous pings Ono performed.

Papers

Taming the Torrent: A Practical Approach to Reducing Cross-ISP Traffic in Peer-to-Peer Systems — SIGCOMM 2008
Drafting Behind Akamai (Travelocity-Based Detouring) — SIGCOMM 2006
Taming the Torrent — USENIX ;login: 2010

Coverage

Ono was covered by Slashdot, El País, and the Chronicle of Higher Education in 2008.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

David R. Choffnes

Contributors

Xian Yi Teng
Nikola Borisov
Shane Ponting

Reusing CDN Measurements — the project Ono was built for

Reusing CDN Measurements

Fri, 01 Sep 2006 00:00:00 +0000

Project Overview

Large content delivery networks spend enormous effort measuring the Internet. To decide which replica should serve a given client, a CDN continuously evaluates paths, load, and latency, and encodes the result in the redirection it hands back. Those redirections are public. Anybody who asks a CDN where to fetch an object gets a small piece of that network view for free.

This project started from the observation that the redirections are a signal in their own right, not just a mechanism for fetching content. If two clients are sent to the same replica at the same time, that says something about where they sit in the network relative to one another. If a client’s redirection changes, something about the path has changed. A CDN’s infrastructure, in other words, doubles as a measurement service with a global vantage point, updated continuously, at no cost to the person reading it — provided the signal can be interpreted correctly.

The work that followed ran that idea through three problems: finding better paths than the ones routing gives you, positioning hosts relative to each other, and choosing peers in a peer-to-peer system.

Detouring

Drafting Behind Akamai showed that CDN redirections can be used to find one-hop detours that beat the default path, without running any measurements of your own. A client watches which replicas a CDN steers it toward, infers path conditions from the pattern, and routes through a peer that the CDN’s own view suggests is well positioned.

SideStep took the same insight further, examining how effective reused measurements are as a substitute for dedicated probing, and packaging the result as a service. We also released DraFTP, an FTP suite built on SideStep, to show the improvement in an application people already used.

Positioning

Relative Network Positioning via CDN Redirections turned the signal toward a different question. Rather than asking which detour is fastest, it asked where hosts sit with respect to one another — using agreement between two clients' redirections as evidence of proximity. This gives a positioning system that needs no landmark infrastructure and no active probing, because the CDN has already done that work.

Peer Selection: Ono

Ono applied the idea where it mattered most commercially. BitTorrent clients pick peers largely at random, which generates enormous volumes of traffic that cross ISP boundaries — expensive for providers, and slower than necessary for users. The prevailing assumption was that these two interests were opposed, and that reducing cross-ISP traffic meant accepting worse performance.

Ono showed otherwise. By using CDN redirections to identify peers that were close in the network, it biased peer selection toward nearby peers, and both numbers improved: less traffic crossed ISP boundaries, and downloads got faster. This was the argument in Taming the Torrent, and it landed in the middle of an active dispute about ISPs throttling peer-to-peer traffic.

Ono shipped as a plugin for the Vuze/Azureus BitTorrent client, which put it in the hands of ordinary users rather than a testbed.

Software

Ono — a Vuze/Azureus plugin implementing CDN-based peer selection. Over 100,000 users; cut latency on selected paths by two orders of magnitude and raised download rates by up to 207%.
SideStep and DraFTP — detouring by reusing CDN measurements, plus an open-source FTP suite built on it. Improved performance more than half the time, and never made it worse.

Papers

Drafting Behind Akamai (Travelocity-Based Detouring) — SIGCOMM 2006
Relative Network Positioning via CDN Redirections — ICDCS 2008
Taming the Torrent: A Practical Approach to Reducing Cross-ISP Traffic in Peer-to-Peer Systems — SIGCOMM 2008
On the Effectiveness of Measurement Reuse for Performance-Based Detouring — INFOCOM 2009
Taming the Torrent — USENIX ;login: 2010

Coverage

Ono was covered by Slashdot, El País, and the Chronicle of Higher Education in 2008.

People

This was a joint project with Aleksandar Kuzmanovic’s group at Northwestern.

Faculty

Fabián E. Bustamante (Northwestern University)
Aleksandar Kuzmanovic (Northwestern University)

Students

David R. Choffnes
Ao-Jan Su (advised by Aleksandar Kuzmanovic)

AquaLab Research Group

NUPastry and NUScribe

Thu, 01 Sep 2005 00:00:00 +0000

Context

The work was motivated by delivering high-quality streaming media to an effectively unlimited number of peers over a purely cooperative model. Multicast suits that class of application because it decouples the size of the receiver set from the state kept at any single node, and avoids redundant transmission. Application-level approaches put all the multicast functionality in end systems configured into an overlay, keeping most of the benefit of network-layer multicast while avoiding IP multicast’s deployment and scalability problems.

Trees are the natural structure for timely delivery, and they scale well in link stress, overhead and latency. But pushed onto autonomous, unpredictable end systems they suffer three specific problems:

Transiency. End systems leave far more readily than routers, and trees depend on their interior nodes.
Bandwidth constraints near the root. Available bandwidth decreases as one ascends from the leaves, so high-level paths become the bottleneck under load.
Unfair load distribution. A small fraction of interior nodes do all the replication and forwarding while leaves contribute nothing — awkward in a system premised on every peer contributing in exchange for the service.

We proposed three generic techniques against these: co-leaders, alternative leaders that remove single-node dependencies and provide alternative forwarding paths; fat-trees, imported from parallel computing to relieve the bandwidth constraint near the root; and multiple performance-centric balanced trees, which allow fair resource contribution without sacrificing performance.

Those became three reference implementations: Nemo, a structurally resilient overlay; FatNemo, a fat-tree overlay for high-bandwidth multicast; and Magellan, a performance-centric multi-tree overlay maximizing fair load distribution against stated performance expectations.

What Was Released

NUPastry — reference implementation of a resilient distributed hash table
NUScribe — DHT-based multicast, with a work-in-progress NUSplitStream providing many of the proposed SplitStream features

They came with documentation for the supporting pieces: a DHT API, the Heimdall network library, a multicast API, and a common API layer.

Software was distributed to users who registered on the lab site, on the condition that any paper using it cite the corresponding publication and that a copy be sent to us on publication. It is no longer available. Wide-area experiments were run with the Nixes tool set.

Papers

Resilient Peer-to-Peer Multicast from the Ground Up — NCA 2004
FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree — WCW 2004
Nemo: Resilient Peer-to-Peer Multicast without the Cost — MMCN 2005
FatNemo: Multisource Multicast Overlay Fat-Trees — NSDI 2005 (poster)
Magellan: Performance-based, Cooperative Multicast — WCW 2005
The Feasibility of DHT-based Streaming Multicast — MASCOTS 2005
Resilience in Overlay Multicast Protocols — MASCOTS 2006

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Peter Dinda (Northwestern University)
Dong Lu (Northwestern University)

Resilient Overlays and Multicast — the project these implementations belong to

FatNemo and FatScribe

Fri, 01 Oct 2004 00:00:00 +0000

Motivation and Approach

For streaming-oriented applications, where timely delivery is the key requirement, trees are generally the preferred overlay structure. But when the application is bandwidth-demanding — video conferencing, multi-party games — basic trees run into a structural problem. Bandwidth availability decreases monotonically as one ascends from the leaves, so under high load the paths near the root become the bottleneck and dominate delivery latency. Once those links are loaded, packets are buffered, then dropped.

We imported the concept of fat-trees from parallel architectures into overlay networks to address exactly that constraint. Fat-trees widen as they approach the root, giving the interior the capacity that traffic concentration demands.

We implemented two protocols on this idea: FatNemo, for performance-centric overlay multicast, and FatScribe, for DHT-based systems.

Results

Through simulation and wide-area experimentation — against each other and against alternative protocols — the fat-tree approach was shown to:

lower the forwarding responsibility of individual nodes, increasing scalability to match the demands of high-bandwidth multicast
reduce the height of the forwarding tree, significantly shortening delivery latency
improve robustness to node transiency, by increasing path diversity in the overlay

That last point matters most in practice. An overlay is built from machines that leave without warning, and a structure with more distinct paths degrades more gracefully when they do.

Availability

Reference implementations were distributed to users who registered on the lab site, with the requirement that any paper using the software cite the corresponding publication. They are no longer available. Wide-area deployment used the Nixes tool set.

Papers

FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree — WCW 2004
FatNemo: Multisource Multicast Overlay Fat-Trees — NSDI 2005 (poster)
The Feasibility of DHT-based Streaming Multicast — MASCOTS 2005

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Peter Dinda (Northwestern University)
Dong Lu (Northwestern University)

Resilient Overlays and Multicast — the project FatNemo was built for

Resilient Overlays and Multicast

Sun, 01 Jun 2003 00:00:00 +0000

Project Overview

IP multicast was never deployed widely enough to carry streaming media across the Internet, so the job fell to overlays built out of the participants themselves. That trade is attractive — no infrastructure required — and it creates a hard problem: the network you are building on is made of ordinary machines that leave without warning. A structure that delivers efficiently when everyone stays put can collapse when a fraction of its interior nodes disappear at once.

This project ran for several years on the question of what overlay structure actually survives churn, and produced both protocols and a body of comparative measurement about how the alternatives behave when conditions stop being convenient.

Nemo

Nemo’s argument was that resilience need not be bought with overhead. Prevailing approaches improved reliability by adding redundancy — more paths, more copies, more maintenance traffic — and paid for it in bandwidth and complexity. Nemo used co-leaders to keep delivery going when interior nodes fail, achieving resilience comparable to far more expensive schemes at a fraction of the cost. Nemo: Resilient Peer-to-Peer Multicast without the Cost is the paper that carries the argument.

FatNemo

FatNemo borrowed a structure from parallel computing. Leiserson’s fat-trees widen as they approach the root, giving interior links the capacity that traffic concentration demands — exactly the property a multi-source multicast overlay needs, where conventional trees choke near the top. FatNemo builds that shape out of peers, sizing the overlay’s interior to the bandwidth it must carry rather than assuming uniform links.

DHT-Based Variants and Comparison

Alongside the purpose-built protocols we examined whether general-purpose structured overlays could do the job, releasing reference implementations of NUPastry and NUScribe and studying the feasibility of DHT-based streaming multicast. The comparative strand ran throughout the project — Resilience in Overlay Multicast Protocols, A Comparison of Resilient Overlay Multicast Approaches, and Structured and Unstructured Overlays Under the Microscope — and fed a related question about how much structure an overlay should have at all when churn is the normal condition rather than the exception.

Software

Nemo — reference implementation of a structurally resilient, performance-centric overlay multicast protocol for streaming applications. Its own page is long gone; an Internet Archive snapshot survives.
FatNemo and FatScribe — fat-tree overlays addressing the bandwidth constraints of conventional tree multicast.
NUPastry and NUScribe — a resilient distributed hash table and DHT-based multicast, with a work-in-progress NUSplitStream.
The Nixes Tool Set — the PlanetLab scripts these wide-area experiments were deployed with.

Papers

Nemo: Resilient Peer-to-Peer Multicast — Northwestern CS technical report, 2003
Resilient Peer-to-Peer Multicast from the Ground Up — NCA 2004
FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree — WCW 2004
Nemo: Resilient Peer-to-Peer Multicast without the Cost — MMCN 2005
FatNemo: Multisource Multicast Overlay Fat-Trees — NSDI 2005 (poster)
The Feasibility of DHT-based Streaming Multicast — MASCOTS 2005
Magellan: Performance-based, Cooperative Multicast — WCW 2005
Elders Know Best: Handling Churn in Less Structured P2P Systems — IEEE P2P 2005
Resilience in Overlay Multicast Protocols — MASCOTS 2006
Structured and Unstructured Overlays Under the Microscope — USENIX ATC 2006
Designing Less-structured P2P Systems for the Expected High Churn — IEEE/ACM Transactions on Networking, 2007
A Comparison of Resilient Overlay Multicast Approaches — IEEE JSAC, 2007

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Dong Lu (Northwestern University)
Peter Dinda (Northwestern University)

AquaLab Research Group

Peer-to-Peer | Aqualab - Northwestern University

Measurement and Experimentation from the Edge

Project Overview

Two Projects

Software

Papers

Coverage

People

Related Links

Dasu

The Short Story

Why Continuous, and Why From the Edge

Distribution

Why the Name

Papers

Coverage

People

Related Links

Privacy in Peer-to-Peer Systems

Project Overview

SwarmScreen

Software

Papers

Coverage

People

Related Links

SwarmScreen

The Short Story

The Attack

The Defence

Why the Name

Availability

Papers

Coverage

People

Related Links

NEWS

The Short Story

Why the Edge

Distribution

Papers

Coverage

People

Related Links

Ono

Motivation and Approach

Results

Why the Name

Why Not the Alternatives

Availability

Papers

Coverage

People

Related Links

Reusing CDN Measurements

Project Overview

Detouring

Positioning

Peer Selection: Ono

Software

Papers

Coverage

People

Related Links

NUPastry and NUScribe

Context

What Was Released

Papers

People

Related Links

FatNemo and FatScribe

Motivation and Approach

Results

Availability

Papers

People

Related Links

Resilient Overlays and Multicast

Project Overview

Nemo