What: NSF/FCC Workshop - Tracking Quality of Experience in the Internet

Where: Princeton U. Campus, Princeton, NJ

When: October 21-22, 2015

The term Quality of Experience, or QoE, is used to describe the subjective assessment by a user of the experience of using a specific application. Network engineers have in the past focused on Quality of Service: performance metrics such as throughput, delay and jitter, packet loss, and the like. But different performance as measured by QoS parameters only matters if it affects the consequential experience of users, as they attempt to use one or another application over the net. QoE impairments, which might include rebuffering for streaming video or excessive round trip latency for real time speech, are what really matter to the user.

The purpose of this workshop is to understand the state of research with respect to QoE, and specifically to contemplate a community agenda to integrate ongoing threads of QoE research into a collaboration, in order to measure and analyze the state of QoE in the current Internet and identify and localize the causes of QoE impairments. Localization of the causes of QoE impairments is a complex problem, since the impairments may arise in one or another region of the network, in the home network of the user, in the computer of the user, in servers that are part of the application, or in supporting services such as the DNS. For this reason, identification and localization of QoE impairments may require collaboration among researchers who do different sorts of measurement of network and service performance. The objective of the workshop is to attempt to establish one or more frameworks for this sort of collaboration.

The workshop is organized by Fabián E. Bustamante (Northwestern U), David Clark (MIT), and Nick Feamster (Princeton U.). The plan is two have two meetings, the first of which will be held at the Princeton U. Campus, in Princeton, NJ on October 21-22, 2015.

Workshop Chairs

Fabián E. Bustamante, Northwestern U.
David Clark, MIT
Nick Feamster, Princeton U.

Program Committee

Padma Krishnaswamy, FCC
Jason Livingood, Comcast
Richard Mortier, Cambridge U.
Andrea Soppera, British Telecom
Renata Teixeira, INRIA

We are soliciting participation from people who are already doing research that relates to this objective, which might include characterization of application-specific QoE models and metrics, measurement of relevant aspects of underlying performance, curation and analysis of performance data, development and deployment of measurement infrastructures and the like. The goal of the workshop is not to present our latest research results to each other, but to understand the overall current state of the relevant research and the potential for collaboration to further our collective understanding of QoE.

The plan is to hold two workshops, and we expect that participants will attend both. The first is this fall (see dates below) and the second tentatively next Winter, in Washington, DC. The hope is that by having a second workshop to follow up on the discussion that occurs at the first, we can make real progress in defining a research agenda and a framework for community collaboration.

If you are interested in attending, please prepare a short (1-3 pages) white paper that describes the scope of your current research and how it might fit into a larger framework or collaboration around the issues of QoE. These white papers will be used to select a balanced set of attendees, and will be circulated to the attendees prior to the workshop to give all the attendees a common understanding of the state of research. White papers and meeting summaries will be posted on a workshop web page as part of the workshop results.

Please use the following link for SUBMISSIONS. We are using HotCRP; you will have to create an account first.

Important dates

  • White papers due: September 4, 2015 (11:59PM EST)
  • Notifications: September 20, 2015
  • Workshop - First meeting: October 21-22, 2015
  • Workshop - Second meeting: TBD

Travel support

Since we are soliciting participation from people already active in the field, we are hoping that most will be able to cover the cost of attending from their current sources of support. NSF has also made available limited support for travel grants for qualified participants who would otherwise not be able to attend.

List of participants

  • Matthew Andrews, Alcatel-Lucent Bell Labs
  • Peter Arzberger, NSF
  • Fred Baker, CISCO
  • Aruna Balasubramanian, Stonny Brook U.
  • Suman Banerjee, U. Wisconsin Madison
  • Steve Bauer, MIT
  • John Brassil, NSF
  • Fabián E. Bustamante, Northwestern U.
  • David Choffnes, Northeastern U.
  • kc claffy, CAIDA/USCD
  • David Clark, MIT
  • Amogh Dhamdhere, CAIDA/UCSD
  • Constantine Dovrolis, Georgia Tech
  • Nick Feamster, Princeton U.
  • Darleen Fisher, NSF
  • Kyle Jamieson, Princeton U.
  • Walter Johnston, FCC
  • Scott Jordan, FCC
  • Yana Kane-Esrig, Comcast
  • Padma Krishnaswamy, FCC
  • William Lehr, MIT
  • Yong Liu, NYU
  • Jason Livingood, Comcast
  • Z. Morley Mao, U. Michigan
  • Keith Marzullo, NITRD
  • Maria Papadoupouli, U. of Crete, ICS-FORTH
  • Henning Schulzrinne, Columbia U.
  • Vyas Sekar, CMU
  • Srikanth Sundaresan, ICSI
  • M. Zubair Shafiq, U. of Iowa
  • Ramesh Sitaraman, U. of Massachusetts, Amherst
  • Lea Skorin-Kapov, U. of Zagreb
  • Srikanth Sundaresan, ICSI
  • Renata Teixeira, INRA
  • Walter Willinger, NIKSUN
  • Hui Zhang, CMU/Conviva
  • Zhi-Li Zhang, U. of Minessota
  • Thyaga Nandagopal, NSF

Background and format

The purpose of this workshop is to understand the state of research with respect to QoE, and specifically to contemplate a community agenda to integrate ongoing threads of QoE research into a collaboration, in order to measure and analyze the state of QoE in the current Internet, identify and localize the causes of QoE impairments, and find ways to improve QoE for the user.

There are a number of aspects of QoE. This agenda takes a top-down approach in organizing our sessions, starting with user measurement studies and ending with network measurement infrastructure.

We have organized each session as a series of short talks followed by time for discussion. We are asking each speaker to limit themselves to 10 minutes. Based on our experience at other workshops, we believe it is possible to focus on the most important aspects of an effort and describe the key conclusions in 10 minutes, given that the audience will be fully aware of the background and context of the specific research.

You will see that some sessions have only one or zero papers. Several of the papers addressed more than one topic area. if presenters feel they would be better suited to another session, please speak up. We will combine sessions as appropriate if and as we get comments from our presenters.

We did not ask every one of our attendees to give a presentation. We focused on the papers that most directly addressed the high-level goals of the workshop. We assume that all participants will be involved in the discussion, which is the most important part of the event.

Here is a link to the Final report.


Day 1
08:30AM – 09:00AM Continental Breakfast
09:00AM – 09:10AM Introductions - Goals and meeting format
09:10AM – 10:35AM Session 1: Measuring QoE Assesment of Real Users
There has been a lot of laboratory research where users interact with specific applications while they are subjected to various sorts of controlled impairments. The state of the art seems to be shifing to field trials where the QoE of the user is assessed indirectly (e.g., by abandonment of the app) or by a survey or "I'm annoyed" button. This session reviews work in this area.
10:35AM – 10:50AMBreak
10:50AM – 11:45AM Session 2: Correlating QoE to observed operating parameters
Assuming that there is some means to estimate the current QoE, the next line of research is to correlate this with various observed operating parameters, both at the application and system level.
11:45AM – 12:45PMLunch
12:45PM – 1:40PM Session 3: Application design and QoE
These presentations focus on application designs and how they can affect QoE.
01:40PM – 2:50PM Session 4: Cross-layer adaptation for QoE
Assuming that the application designer has tools to assess the current QoE, the app can take adaptive steps to compensate for the assumed cause of the impairment. An important example is rate adaptation for video or codec adaptation for VoIP. These presentations focuses on cross-layer adaptation.
02:50PM – 03:05PMBreak
03:05PM – 04:15PM Session 5: Measurement of network performance
Measurement of network performance (QoS) is a necessary component of understanding and improving QoE. These presentations focus on measurement of QoS.
04:15PM – 05:10PM Session 6: Economics of QoE
05:10PM – ... Adjourn - Drinks and dinner

Day 2
The second day will transition from structured sessions with presentations to a series of group discussions looking at the higher-level goals of the workshop. The initial sessions here are intended to stimulate that transition.
08:30AM – 09:00AM Continental Breakfast
09:00AM – 09:30AM Session 7: Localization and coordination
To detect that there is an impairment to QoE does not give much guidance as to how to remedy it. A necessary step is to localize the problem. As well, coordination among several sorts of measurement may be necessary to achieve that localization and identification of root cause.
09:30AM – 10:40AM Session 8: Measurement infrastructure
To perform network performance measurement, some sort of measurement infrastructure is necessary. This session focuses on this challenge.
10:40AM – 11:00AMBreak
11:00AM – 12:00PM Open discussion
  • What sort of results might come from various forms of collaboration?
  • How can different sorts of data be combined and correlated togenerate a more complete picture of the locus of impairments?
  • Do we need new sorts of measurements?
  • Are there specific research efforts that we should identify?
  • Is there value in a second workshop to further develop an agenda for research and collaboration?
12:00PM – 01:00PMLunch

Additional papers

Nothing yet

Venue and Accommodation

The workshop's first meeting will be held at the Princeton U. Campus, in the Frist Campus Center, Multipurpose Room B.

For logistic information, please visit Workshop Logistics.