Crowdsourcing | Aqualab - Northwestern University

Measurement Platforms for Mobile Devices

Wed, 01 Feb 2012 00:00:00 +0000

Project Overview

Studying mobile networks and applications runs into a practical wall: the device. A phone will not let you sit in the middle of its traffic, its battery behaviour cannot be simulated convincingly, and the person holding it has their own plans. Each of those obstacles produced an instrument.

These are tools rather than findings — the apparatus that the measurement studies in our mobile thread depend on.

MBZ: a middlebox on the handset

Traditional middleboxes see traffic at the far end of the first link, by which point the interesting attribution — which app, which library, which permission — is gone. MBZ takes that vantage point and moves it onto the device: an extensible platform that intercepts and can modify traffic before it leaves the handset, or on arrival before it reaches the operating system.

Conceptually it gives a middlebox’s view with the device’s context attached, which is what makes it possible to ask questions about specific applications rather than about aggregate flows. Back in Control describes the design.

BatteryLab: power measurement on real hardware

Energy claims about mobile software are usually made with models, because measuring real draw needs physical instrumentation attached to a real device. BatteryLab is a collaborative platform that shares that instrumentation: devices wired to power monitors, hosted across sites, available remotely so researchers can measure genuine power consumption on hardware they do not own.

Crowd (Soft) Control

The third obstacle is the human. Opportunistic sensing takes measurements wherever participants happen to be, which leaves coverage to chance. Crowd (Soft) Control asked what becomes possible when participants can be gently steered toward where a measurement is needed, and the later HotMobile paper takes up the question of incentives — what actually motivates people to participate, and why one scheme does not fit every population.

Software

Middle-Box Zero (MBZ) — an extensible platform to intercept and modify traffic on the device, before it leaves or before it reaches the OS. GitHub
BatteryLab — a distributed platform for power monitoring of mobile devices, led by Matteo Varvello. Its project site no longer resolves; see the PAM 2022 paper below.

Papers

Crowd (Soft) Control — HotMobile 2012
No One-Size Fits All: Towards a Principled Approach for Incentives in Mobile Crowdsourcing — HotMobile 2014
Back in Control: An Extensible Middle-Box on Your Phone — arXiv, 2020
BatteryLab: A Collaborative Platform for Power Monitoring — PAM 2022

Coverage

Crowd Soft Control was covered by Communications of the ACM, Wired, and Discovery News in 2012.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

John P. Rula
James Newman

Collaborators

Matteo Varvello (Nokia Bell Labs)
Kleomenis Katevas, Mihai Plesa, Hamed Haddadi (Brave Software / Imperial College London)
Narseo Vallina-Rodriguez (IMDEA Networks)
Abbas Razaghpanah (Stony Brook University)
Mark Allman (ICSI)

TrailBlaze Chicago

Sun, 01 Jan 2012 00:00:00 +0000

Motivation and Approach

TrailBlaze Chicago was an app for Android-owning cyclists in the Chicago area — one of the most bike-friendly big cities in America, and one that had worked hard to become so. But a city can only keep improving its network if it knows where new paths are needed, and that information sits with the people already riding.

TrailBlaze recorded and anonymously reported a rider’s tracks, letting them vote for new paths simply by carrying their phone while doing what they already enjoyed. It also gave the rider something back: a dashboard with distance, speed, and a map, plus the ability to review previously reported routes.

The app came 2nd in the Apps 4 Metro Chicago Grand Challenge, and 4th in the Transportation Round — first among the not-for-profit entries.

It was distributed through the Android Market and is no longer available.

TrailBlaze has no paper of its own. The research questions it touches are treated in our crowdsourcing work:

Crowd (Soft) Control — HotMobile 2012
No One-Size Fits All: Towards a Principled Approach for Incentives in Mobile Crowdsourcing — HotMobile 2014

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

John P. Rula

Civic Crowdsourcing Apps

Civic Crowdsourcing Apps

Thu, 01 Sep 2011 00:00:00 +0000

Project Overview

Most of our crowdsourcing work points at the network: users’ own traffic and devices as a source of measurement about the Internet. These two apps pointed the same idea at the city instead, and were built for the Apps 4 Metro Chicago competitions in 2011 and 2012.

The interesting problem in both is not collection but trust. Anyone can file a report; the question is which reports are real, which are duplicates, and which matter enough to act on. That is a validation problem, and it looks a lot like the one in crowdsourced network measurement, where a single client’s view is cheap and unreliable and the value comes from reconciling many of them.

FixIt!

FixIt! crowdsources both the reporting and the validation of issues requiring city attention — broken streetlights, potholes, graffiti. Reports from one user can be confirmed or disputed by others nearby, so the system accumulates evidence rather than just a queue of claims.

It took 5th place in the Apps 4 Metro Chicago Challenge, Community Round.

TrailBlaze Chicago

TrailBlaze Chicago applies the same approach to cycling: planning routes and reporting the current status of bike paths, where conditions change faster than any official dataset can track and the people who know are the ones riding.

It came 2nd in the Apps 4 Metro Chicago Grand Challenge, and 4th in the Transportation Round — first among the not-for-profit entries.

Software

FixIt! — crowdsourced reporting and report-validation of issues requiring city attention.
TrailBlaze Chicago — crowdsourced planning and status reporting for bike paths.

Neither app has a paper of its own. The research questions they touch are treated in our crowdsourcing work:

Crowd (Soft) Control — HotMobile 2012
No One-Size Fits All: Towards a Principled Approach for Incentives in Mobile Crowdsourcing — HotMobile 2014

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

AquaLab Research Group

FixIt!

Thu, 01 Sep 2011 00:00:00 +0000

The Story

FixIt! was an AquaLab project built around collaborative civic engagement, originally developed for the 2011 Apps for Metro Chicago competition. It allowed citizens to report broken streetlights, potholes, graffiti and other public property issues requiring city attention.

Once something was reported, FixIt! unobtrusively asked other users in the area to confirm or refute the request. The result was a curated, crowdsourced list of items for the city to act on — letting it dispatch personnel with more confidence that there was something to fix, and with a sense of which issues were affecting citizens most.

That confirmation step is the whole design. Collecting reports is easy; knowing which ones are real, which are duplicates, and which matter is the part that makes the data usable, and it is the same problem that shows up in crowdsourced network measurement, where one client’s view is cheap and unreliable and the value comes from reconciling many.

It took 5th place in the Apps 4 Metro Chicago Challenge, Community Round.

The App

Reporting, describing, and — the part that mattered — being asked to confirm somebody else’s report.

FixIt! has no paper of its own. The research questions it touches are treated in our crowdsourcing work:

Crowd (Soft) Control — HotMobile 2012
No One-Size Fits All: Towards a Principled Approach for Incentives in Mobile Crowdsourcing — HotMobile 2014

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Civic Crowdsourcing Apps

NEWS

Wed, 07 May 2008 00:00:00 +0000

The Short Story

The goal was to reliably find problems in the network and raise alerts about them. As a user, you want to know you are getting the service you pay for, and to be told quickly when something breaks — particularly when an interruption might entitle you to compensation. For an ISP, the same information localizes and identifies faults faster, so they can be fixed before they generate complaints.

NEWS did this by passively monitoring BitTorrent performance and watching for changes that suggested network trouble. Because a problem can be anywhere, including inside your own home network, NEWS corroborated across multiple users in the same area — the same ISP, or the same country. When enough people saw the same problem in the same place, it raised an alarm.

The approach’s virtue was its cost: monitoring traffic the client was generating anyway made detection extremely lightweight.

NEWS coverage. Detection depended on having enough users inside the same ISP or country to corroborate a suspected problem.

Why the Edge

The Internet increasingly carried services — VoIP, content distribution, IPTV — whose user experience had become a benchmark for providers and users alike. That experience is largely determined by the frequency, duration and severity of network events, so there is a clear need to detect, isolate and explain those events quickly.

Most prior work monitored core networks or probed from research and education networks. That is effective for faults affecting many customers at once, but it misses silent failures — incompatible QoS or ACL settings, for instance — and their effect on individual services. End-to-end monitoring approaches, meanwhile, required active measurement that does not scale to the number of elements at the network’s edge.

NEWS took the other route: push performance monitoring and detection onto the end systems themselves. Crowdsourcing the monitoring meant participating hosts could collectively handle the data volume needed for real-time detection at the scale of millions of monitors, and installing inside or alongside a service allowed immediate, incremental deployment.

Distribution

NEWS installed into the Vuze/Azureus BitTorrent client and reached hundreds of thousands of users. It was released under the GPL, with source included in the jar. Users contributed translations. It is no longer available.

Release history (2008)

24 Oct 2008 — a Java-version problem affecting macOS identified, fix planned for the next release
1 Jul 2008 — version 0.8 mainlined, with numerous bug fixes
24 Jun 2008 — version 0.6 released as a signed plugin for beta testing at scale
19 May 2008 — version 0.2, mature enough for general use
7 May 2008 — first version released

Papers

Crowdsourcing Service-Level Network Event Detection — SIGCOMM 2010
Using the Crowd to Monitor the Cloud: Detecting Network Events from Edge Systems — Northwestern EECS technical report, 2009

Coverage

NEWS was covered by RedOrbit, Slashdot, and The Hindu in 2008.

People

Lead & PI

Fabián E. Bustamante (Northwestern University)

Students

David R. Choffnes

Measurement and Experimentation from the Edge — the wider thread this belongs to
Dasu — the platform that grew out of the same approach

Crowdsourcing | Aqualab - Northwestern University

Measurement Platforms for Mobile Devices

Project Overview

MBZ: a middlebox on the handset

BatteryLab: power measurement on real hardware

Crowd (Soft) Control

Software

Papers

Coverage

People

Related Links

TrailBlaze Chicago

Motivation and Approach

Related Work

People

Related Links

Civic Crowdsourcing Apps

Project Overview

FixIt!

TrailBlaze Chicago

Software

Related Work

People

Related Links

FixIt!

The Story

The App

Related Work

People

Related Links

NEWS

The Short Story

Why the Edge

Distribution

Papers

Coverage

People

Related Links