Please use this identifier to cite or link to this item:
Title: Learning from multiple annotators: distinguishing good from random labelers
Authors: Rodrigues, Filipe 
Pereira, Francisco 
Ribeiro, Bernardete 
Keywords: Multiple annotators; Crowdsourcing; Latent variable models; Expectation–Maximization; Logistic Regression
Issue Date: 1-Sep-2013
Publisher: Elsevier
Citation: RODRIGUES, Filipe; PEREIRA, Francisco; RIBEIRO, Bernardete - Learning from multiple annotators: distinguishing good from random labelers. "Pattern Recognition Letters". ISSN 0167-8655. Vol. 34 Nº. 12 (2013) p. 1428-1436
Serial title, monograph or event: Pattern Recognition Letters
Volume: 34
Issue: 12
Abstract: With the increasing popularity of online crowdsourcing platforms such as Amazon Mechanical Turk (AMT), building supervised learning models for datasets with multiple annotators is receiving an increasing attention from researchers. These platforms provide an inexpensive and accessible resource that can be used to obtain labeled data, and in many situations the quality of the labels competes directly with those of experts. For such reasons, much attention has recently been given to annotator-aware models. In this paper, we propose a new probabilistic model for supervised learning with multiple annotators where the reliability of the different annotators is treated as a latent variable. We empirically show that this model is able to achieve state of the art performance, while reducing the number of model parameters, thus avoiding a potential overfitting. Furthermore, the proposed model is easier to implement and extend to other classes of learning problems such as sequence labeling tasks.
ISSN: 0167-8655
DOI: 10.1016/j.patrec.2013.05.012
Rights: openAccess
Appears in Collections:I&D CISUC - Artigos em Revistas Internacionais
FCTUC Eng.Informática - Artigos em Revistas Internacionais

Files in This Item:
File Description SizeFormat
Learning from Multiple Annotators.pdf1.45 MBAdobe PDFView/Open
Show full item record


checked on May 29, 2020

Citations 5

checked on Dec 2, 2021

Page view(s)

checked on Nov 24, 2021

Download(s) 20

checked on Nov 24, 2021

Google ScholarTM




Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.