Schedule

WORKSHOP AGENDA

09:00-09:10        Introduction

09:10-09:40        A Brief Survey of Machine Learning in Human Computation and Crowdsourcing
Rajarshi Das, IBM Research, USA

9:40-10:20        Crowd IQ: Measuring the Intelligence of Crowdsourcing Platforms
Michal Kosinski, Yoram Bachrach, Gjergji Kasneci, Jurgen Van-Gael, and Thore Graepel
Microsoft Research, UK

10:30-11:00        Coffee break

11:00-11:40        Get another worker? Active crowdlearning with sequential arrivals
James Zou and David Parkes
Harvard University, USA

11:40-12:05        Aggregating Human-Expert Votes using Stacked Generalization
Evgueni Smirnov, Hua Zhang, and Nikolay Nikolaev
Maastricht University, Netherlands & University of London, UK

12:05-12:30        Improving Repeated Labeling for Crowdsourced Data Annotation
Sergiu Goschin, Chris Mesterharm, and Haym Hirsh
Rutgers University, USA

12:30-14:00        Lunch

14:00-14:40       How To Grade a Test Without Knowing the Answers — A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing
Yoram Bachrach, Thore Graepel, Tom Minka, and John Guiver
Microsoft Research, UK

14:40-15:05      Factor-based Regression Models for Forecasting
Chih-Chieh Cheng, Robert Sasseen, Tulay Muezzinoglu, and Richard Rohwer
SRI International, USA

15:05-15:30        Dynamic Estimation of Rater Reliability in Regression Tasks using Multi-Armed Bandit Techniques
Alexey Tarasov, Sarah Jane Delany, and Brian Mac Namee
Dublin Institute of Technology, Ireland

15:30-16:00        Coffee break

16:00-16:25        Crowdsourcing Microtasks Using Multiple Crowds
Ittai Abraham,  Omar Alonso,  Vasilis Kandylas, and Aleksandrs Slivkins
Microsoft Research, USA

16:25-16:50        Suggesting Constraints to Interactive Topic Modeling
Yuening Hu and Jordan Boyd-Graber
University of Maryland, USA

16:50-17:30        General Discussion
All Participants