Multicast | Aqualab - Northwestern University

NUPastry and NUScribe

Thu, 01 Sep 2005 00:00:00 +0000

Context

The work was motivated by delivering high-quality streaming media to an effectively unlimited number of peers over a purely cooperative model. Multicast suits that class of application because it decouples the size of the receiver set from the state kept at any single node, and avoids redundant transmission. Application-level approaches put all the multicast functionality in end systems configured into an overlay, keeping most of the benefit of network-layer multicast while avoiding IP multicast’s deployment and scalability problems.

Trees are the natural structure for timely delivery, and they scale well in link stress, overhead and latency. But pushed onto autonomous, unpredictable end systems they suffer three specific problems:

Transiency. End systems leave far more readily than routers, and trees depend on their interior nodes.
Bandwidth constraints near the root. Available bandwidth decreases as one ascends from the leaves, so high-level paths become the bottleneck under load.
Unfair load distribution. A small fraction of interior nodes do all the replication and forwarding while leaves contribute nothing — awkward in a system premised on every peer contributing in exchange for the service.

We proposed three generic techniques against these: co-leaders, alternative leaders that remove single-node dependencies and provide alternative forwarding paths; fat-trees, imported from parallel computing to relieve the bandwidth constraint near the root; and multiple performance-centric balanced trees, which allow fair resource contribution without sacrificing performance.

Those became three reference implementations: Nemo, a structurally resilient overlay; FatNemo, a fat-tree overlay for high-bandwidth multicast; and Magellan, a performance-centric multi-tree overlay maximizing fair load distribution against stated performance expectations.

What Was Released

NUPastry — reference implementation of a resilient distributed hash table
NUScribe — DHT-based multicast, with a work-in-progress NUSplitStream providing many of the proposed SplitStream features

They came with documentation for the supporting pieces: a DHT API, the Heimdall network library, a multicast API, and a common API layer.

Software was distributed to users who registered on the lab site, on the condition that any paper using it cite the corresponding publication and that a copy be sent to us on publication. It is no longer available. Wide-area experiments were run with the Nixes tool set.

Papers

Resilient Peer-to-Peer Multicast from the Ground Up — NCA 2004
FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree — WCW 2004
Nemo: Resilient Peer-to-Peer Multicast without the Cost — MMCN 2005
FatNemo: Multisource Multicast Overlay Fat-Trees — NSDI 2005 (poster)
Magellan: Performance-based, Cooperative Multicast — WCW 2005
The Feasibility of DHT-based Streaming Multicast — MASCOTS 2005
Resilience in Overlay Multicast Protocols — MASCOTS 2006

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Peter Dinda (Northwestern University)
Dong Lu (Northwestern University)

Resilient Overlays and Multicast — the project these implementations belong to

FatNemo and FatScribe

Fri, 01 Oct 2004 00:00:00 +0000

Motivation and Approach

For streaming-oriented applications, where timely delivery is the key requirement, trees are generally the preferred overlay structure. But when the application is bandwidth-demanding — video conferencing, multi-party games — basic trees run into a structural problem. Bandwidth availability decreases monotonically as one ascends from the leaves, so under high load the paths near the root become the bottleneck and dominate delivery latency. Once those links are loaded, packets are buffered, then dropped.

We imported the concept of fat-trees from parallel architectures into overlay networks to address exactly that constraint. Fat-trees widen as they approach the root, giving the interior the capacity that traffic concentration demands.

We implemented two protocols on this idea: FatNemo, for performance-centric overlay multicast, and FatScribe, for DHT-based systems.

Results

Through simulation and wide-area experimentation — against each other and against alternative protocols — the fat-tree approach was shown to:

lower the forwarding responsibility of individual nodes, increasing scalability to match the demands of high-bandwidth multicast
reduce the height of the forwarding tree, significantly shortening delivery latency
improve robustness to node transiency, by increasing path diversity in the overlay

That last point matters most in practice. An overlay is built from machines that leave without warning, and a structure with more distinct paths degrades more gracefully when they do.

Availability

Reference implementations were distributed to users who registered on the lab site, with the requirement that any paper using the software cite the corresponding publication. They are no longer available. Wide-area deployment used the Nixes tool set.

Papers

FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree — WCW 2004
FatNemo: Multisource Multicast Overlay Fat-Trees — NSDI 2005 (poster)
The Feasibility of DHT-based Streaming Multicast — MASCOTS 2005

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Peter Dinda (Northwestern University)
Dong Lu (Northwestern University)

Resilient Overlays and Multicast — the project FatNemo was built for

Resilient Overlays and Multicast

Sun, 01 Jun 2003 00:00:00 +0000

Project Overview

IP multicast was never deployed widely enough to carry streaming media across the Internet, so the job fell to overlays built out of the participants themselves. That trade is attractive — no infrastructure required — and it creates a hard problem: the network you are building on is made of ordinary machines that leave without warning. A structure that delivers efficiently when everyone stays put can collapse when a fraction of its interior nodes disappear at once.

This project ran for several years on the question of what overlay structure actually survives churn, and produced both protocols and a body of comparative measurement about how the alternatives behave when conditions stop being convenient.

Nemo

Nemo’s argument was that resilience need not be bought with overhead. Prevailing approaches improved reliability by adding redundancy — more paths, more copies, more maintenance traffic — and paid for it in bandwidth and complexity. Nemo used co-leaders to keep delivery going when interior nodes fail, achieving resilience comparable to far more expensive schemes at a fraction of the cost. Nemo: Resilient Peer-to-Peer Multicast without the Cost is the paper that carries the argument.

FatNemo

FatNemo borrowed a structure from parallel computing. Leiserson’s fat-trees widen as they approach the root, giving interior links the capacity that traffic concentration demands — exactly the property a multi-source multicast overlay needs, where conventional trees choke near the top. FatNemo builds that shape out of peers, sizing the overlay’s interior to the bandwidth it must carry rather than assuming uniform links.

DHT-Based Variants and Comparison

Alongside the purpose-built protocols we examined whether general-purpose structured overlays could do the job, releasing reference implementations of NUPastry and NUScribe and studying the feasibility of DHT-based streaming multicast. The comparative strand ran throughout the project — Resilience in Overlay Multicast Protocols, A Comparison of Resilient Overlay Multicast Approaches, and Structured and Unstructured Overlays Under the Microscope — and fed a related question about how much structure an overlay should have at all when churn is the normal condition rather than the exception.

Software

Nemo — reference implementation of a structurally resilient, performance-centric overlay multicast protocol for streaming applications. Its own page is long gone; an Internet Archive snapshot survives.
FatNemo and FatScribe — fat-tree overlays addressing the bandwidth constraints of conventional tree multicast.
NUPastry and NUScribe — a resilient distributed hash table and DHT-based multicast, with a work-in-progress NUSplitStream.
The Nixes Tool Set — the PlanetLab scripts these wide-area experiments were deployed with.

Papers

Nemo: Resilient Peer-to-Peer Multicast — Northwestern CS technical report, 2003
Resilient Peer-to-Peer Multicast from the Ground Up — NCA 2004
FatNemo: Building a Resilient Multi-Source Multicast Fat-Tree — WCW 2004
Nemo: Resilient Peer-to-Peer Multicast without the Cost — MMCN 2005
FatNemo: Multisource Multicast Overlay Fat-Trees — NSDI 2005 (poster)
The Feasibility of DHT-based Streaming Multicast — MASCOTS 2005
Magellan: Performance-based, Cooperative Multicast — WCW 2005
Elders Know Best: Handling Churn in Less Structured P2P Systems — IEEE P2P 2005
Resilience in Overlay Multicast Protocols — MASCOTS 2006
Structured and Unstructured Overlays Under the Microscope — USENIX ATC 2006
Designing Less-structured P2P Systems for the Expected High Churn — IEEE/ACM Transactions on Networking, 2007
A Comparison of Resilient Overlay Multicast Approaches — IEEE JSAC, 2007

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

Collaborators

Dong Lu (Northwestern University)
Peter Dinda (Northwestern University)

AquaLab Research Group

Multicast | Aqualab - Northwestern University

NUPastry and NUScribe

Context

What Was Released

Papers

People

Related Links

FatNemo and FatScribe

Motivation and Approach

Results

Availability

Papers

People

Related Links

Resilient Overlays and Multicast

Project Overview

Nemo

FatNemo

DHT-Based Variants and Comparison

Software

Papers

People

Related Links