Architecting Real-Time Crowd-Powered Systems
DOI:
https://doi.org/10.15346/hc.v1i1.5Keywords:
real-time crowdsourcing, human computation, system architecturesAbstract
Human computation allows computer systems to leverage human intelligence in computational processes. While it has primarily been used for tasks that are not time-sensitive, recent systems use crowdsourcing to get on-demand, real-time, and even interactive results. In this paper, we present techniques for building real-time crowdsourcing systems, and then discuss how and when to use them. Our goal is to provide system builders with the tools and insights they need to replicate the success of modern systems in order to further explore this new space.References
Words per minute. http://en.wikipedia.org/wiki/Words_per_minute. (????). Accessed: 2014-06-05.
Archak, N and Sundararajan, A. (2009). Optimal Design of Crowdsourcing Contests.. In ICIS. 200.
Arrow, K. J. (2012). Social choice and individual values. Vol. 12. Yale university press.
Bernstein, D. S, Givan, R, Immerman, N, and Zilberstein, S. (2002). The complexity of decentralized control of Markov decision
processes. Mathematics of operations research 27, 4 (2002), 819–840.
Bernstein, M. S, Brandt, J, Miller, R. C, and Karger, D. R. (2011). Crowds in Two Seconds: Enabling Realtime Crowd-powered
Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New
York, NY, USA, 33–42. DOI:http://dx.doi.org/10.1145/2047196.2047201
Bernstein, M. S, Karger, D. R, Miller, R. C, and Brandt, J. (2012). Analytic Methods for Optimizing Realtime Crowdsourcing. CoRR
abs/1204.2995 (2012).
Bernstein, M. S, Little, G, Miller, R. C, Hartmann, B, Ackerman, M. S, Karger, D. R, Crowell, D, and Panovich, K. (2010). Soylent:
A Word Processor with a Crowd Inside. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and
Technology (UIST ’10). ACM, New York, NY, USA, 313–322. DOI:http://dx.doi.org/10.1145/1866029.1866078
Bigham, J. P, Jayant, C, Ji, H, Little, G, Miller, A, Miller, R. C, Miller, R, Tatarowicz, A, White, B, White, S, and Yeh, T. (2010).
VizWiz: Nearly Real-time Answers to Visual Questions. In Proceedings of the 23Nd Annual ACM Symposium on User Interface
Software and Technology (UIST ’10). ACM, New York, NY, USA, 333–342. DOI:http://dx.doi.org/10.1145/1866029.1866080
Bigham, J. P and Ladner, R. E. (2011). What the Disability Community Can Teach Us About Interactive Crowdsourcing. interactions
, 4 (July 2011), 78–81. DOI:http://dx.doi.org/10.1145/1978822.1978838
Bigham, J. P and Lasecki, W. S. (2014). Crowd Storage: Storing Information on Existing Memories. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 601–604. DOI:http://dx.doi.org/10.
/2556288.2557159
Borish, M, Cordar, A, Foster, A, Kim, T, Murphy, J, and Lok, B. (2014). Utilizing Real-time Human-Assisted Virtual Humans to
Increase Real-world Interaction Empathy. In Kansei Engineering & Emotion Research (KEER ’14). 15.
Boudreau, K. J, Lacetera, N, and Lakhani, K. R. (2011). Incentives and problem uncertainty in innovation contests: An empirical
analysis. Management Science 57, 5 (2011), 843–863.
Boutilier, C. (1999). Sequential optimality and coordination in multiagent systems. In IJCAI, Vol. 99. 478–485.
Boutsis, I and Kalogeraki, V. (2013). Crowdsourcing under Real-Time Constraints. In Parallel Distributed Processing (IPDPS), 2013
IEEE 27th International Symposium on. 753–764. DOI:http://dx.doi.org/10.1109/IPDPS.2013.84
Callison-Burch, C. (2009). Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. In Proceedings
of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1 (EMNLP ’09).
Association for Computational Linguistics, Stroudsburg, PA, USA, 286–295. http://dl.acm.org/citation.cfm?id=1699510.1699548
Carpenter, B. (2011). A Hierarchical Bayesian Model of Crowdsourced Relevance Coding.. In TREC.
Chawla, S, Hartline, J. D, and Sivan, B. (2012). Optimal crowdsourcing contests. In SODA. 856–868.
Chilton, L. B, Sims, C. T, Goldman, M, Little, G, and Miller, R. C. (2009). Seaweed: A Web Application for Designing Economic
Games. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’09). ACM, New York, NY, USA,
–35. DOI:http://dx.doi.org/10.1145/1600150.1600162
Cooper, S, Khatib, F, Treuille, A, Barbero, J, Lee, J, Beenen, M, Leaver-Fay, A, Baker, D, Popovi´c, Z, and others, . (2010). Predicting
protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756–760.
Dai Peng, M and Weld, D. S. (2011). Artificial intelligence for artificial artificial intelligence. In Twenty-Fifth AAAI Conference on
Artificial Intelligence (AAAI 2011).
Dalvi, N, Dasgupta, A, Kumar, R, and Rastogi, V. (2013). Aggregating crowdsourced binary ratings. In Proceedings of the 22nd
international conference on World Wide Web. International World Wide Web Conferences Steering Committee, 285–294.
DiPalantino, D and Vojnovic, M. (2009). Crowdsourcing and all-pay auctions. In Proceedings of the 10th ACM conference on Electronic
commerce. ACM, 119–128.
Dow, S, Kulkarni, A, Klemmer, S, and Hartmann, B. (2012). Shepherding the Crowd Yields Better Work. In Proceedings of the
ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12). ACM, New York, NY, USA, 1013–1022. DOI:
http://dx.doi.org/10.1145/2145204.2145355
Faliszewski, P, Hemaspaandra, E, and Hemaspaandra, L. A. (2010). Using complexity to protect elections. Commun. ACM 53, 11
(2010), 74–82.
Forlines, C, Miller, S, Guelcher, L, and Bruzzi, R. (2014). Crowdsourcing the Future: Predictions Made with a Social Network. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 3655–
DOI:http://dx.doi.org/10.1145/2556288.2556967
Goldman, C. V and Zilberstein, S. (2004). Decentralized control of cooperative systems: Categorization and complexity analysis. J.
Artif. Intell. Res.(JAIR) 22 (2004), 143–174.
Heidari, H and Kearns, M. (2013). Depth-Workload Tradeoffs for Workforce Organization. In First AAAI Conference on Human
Computation and Crowdsourcing.
Hemaspaandra, E, Hemaspaandra, L. A, and Rothe, J. (2012). Online Voter Control in Sequential Elections.. In ECAI. 396–401.
Hemaspaandra, E, Hemaspaandra, L. A, and Rothe, J. (2014). The complexity of online manipulation of sequential elections. J.
Comput. System Sci. 80, 4 (2014), 697–710.
Ho, C.-J, Jabbari, S, and Vaughan, J. W. (2013). Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th
International Conference on Machine Learning (ICML-13). 534–542.
Ho, C.-J and Vaughan, J. W. (2012). Online Task Assignment in Crowdsourcing Markets.. In AAAI.
Horton, J. J and Chilton, L. B. (2010). The Labor Economics of Paid Crowdsourcing. In Proceedings of the 11th ACM Conference on
Electronic Commerce (EC ’10). ACM, New York, NY, USA, 209–218. DOI:http://dx.doi.org/10.1145/1807342.1807376
Ipeirotis, P. G. (2010). Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 17,
(2010), 16–21.
Jung, H. J and Lease, M. (2013). Crowdsourced Task Routing via Matrix Factorization. CoRR abs/1310.5142 (2013).
Kamar, E, Hacker, S, and Horvitz, E. (2012). Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings
of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS ’12). International
Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467–474. http://dl.acm.org/citation.cfm?id=2343576.
Kamar, E and Horvitz, E. (2012). Incentives for truthful reporting in crowdsourcing. In Proceedings of the 11th International Conference
on Autonomous Agents and Multiagent Systems-Volume 3. International Foundation for Autonomous Agents and Multiagent
Systems, 1329–1330.
Karger, D. R, Oh, S, and Shah, D. (2011). Iterative learning for reliable crowdsourcing systems. In Advances in neural information
processing systems. 1953–1961.
Kittur, A, Nickerson, J. V, Bernstein, M, Gerber, E, Shaw, A, Zimmerman, J, Lease, M, and Horton, J. (2013). The Future of Crowd
Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ’13). ACM, New York, NY, USA,
–1318. DOI:http://dx.doi.org/10.1145/2441776.2441923
Kittur, A, Smus, B, Khamkar, S, and Kraut, R. E. (2011). CrowdForge: Crowdsourcing Complex Work. In Proceedings of the 24th
Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY, USA, 43–52. DOI:
http://dx.doi.org/10.1145/2047196.2047202
Kuhn, H. (1953). Extensive Games and the Problem of Information, Contributions to the Theory of Games II, Kuhn, HW and AW
Tucker, eds. 193-216. (1953).
Kulkarni, A, Can, M, and Hartmann, B. (2012). Collaboratively Crowdsourcing Workflows with Turkomatic. In Proceedings of the
ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ’12). ACM, New York, NY, USA, 1003–1012. DOI:
http://dx.doi.org/10.1145/2145204.2145354
Lasecki, Wand Bigham, J. (2012)a. Self-correcting Crowds. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems
(CHI EA ’12). ACM, New York, NY, USA, 2555–2560. DOI:http://dx.doi.org/10.1145/2212776.2223835
Lasecki, W, Miller, C, Sadilek, A, Abumoussa, A, Borrello, D, Kushalnagar, R, and Bigham, J. (2012). Real-time Captioning by
Groups of Non-experts. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST
’12). ACM, New York, NY, USA, 23–34. DOI:http://dx.doi.org/10.1145/2380116.2380122
Lasecki, W. S and Bigham, J. P. (2012)b. Online Quality Control for Real-time Crowd Captioning. In Proceedings of the 14th
International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’12). ACM, New York, NY, USA, 143–150.
DOI:http://dx.doi.org/10.1145/2384916.2384942
Lasecki, W. S, Gordon, M, Koutra, D, Jung, M, Dow, S, and Bigham, J. P. (2014). Glance: Rapidly Coding Behavioral Video with the
Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST ’14). ACM, New
York, NY, USA, 12.
Lasecki, W. S, Kamar, E, and Bohus, D. (2013)a. Conversations in the Crowd: Collecting Data for Task-Oriented Dialog Learning. In
First AAAI Conference on Human Computation and Crowdsourcing.
Lasecki, W. S, Miller, C. D, and Bigham, J. P. (2013)b. Warping Time for More Effective Real-time Crowdsourcing. In Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, 2033–2036. DOI:
http://dx.doi.org/10.1145/2470654.2466269
Lasecki, W. S, Murray, K. I, White, S, Miller, R. C, and Bigham, J. P. (2011). Real-time Crowd Control of Existing Interfaces. In
Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY,
USA, 23–32. DOI:http://dx.doi.org/10.1145/2047196.2047200
Lasecki, W. S, Song, Y. C, Kautz, H, and Bigham, J. P. (2013)a. Real-time Crowd Labeling for Deployable Activity Recognition.
In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ’13). ACM, New York, NY, USA,
–1212. DOI:http://dx.doi.org/10.1145/2441776.2441912
Lasecki, W. S, Thiha, P, Zhong, Y, Brady, E, and Bigham, J. P. (2013)b. Answering Visual Questions with Conversational Crowd
Assistants. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’13).
ACM, New York, NY, USA, Article 18, 8 pages. DOI:http://dx.doi.org/10.1145/2513383.2517033
Lasecki, W. S, Weingard, L, Ferguson, G, and Bigham, J. P. (2014). Finding Dependencies Between Actions Using the Crowd. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 3095–
DOI:http://dx.doi.org/10.1145/2556288.2557176
Lasecki, W. S, Wesley, R, Nichols, J, Kulkarni, A, Allen, J. F, and Bigham, J. P. (2013). Chorus: A Crowd-powered Conversational
Assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New
York, NY, USA, 151–162. DOI:http://dx.doi.org/10.1145/2501988.2502057
Lasecki, W. S, White, S, Murray, K. I, and Bigham, J. P. (2012). Crowd Memory: Learning in the Collective. In Proceedings of
Collective Intelligence. 8.
Lee, J, Kladwang, W, Lee, M, Cantu, D, Azizyan, M, Kim, H, Limpaecher, A, Yoon, S, Treuille, A, and Das, R. (2014). RNA design
rules from a massive open laboratory. Proceedings of the National Academy of Sciences 111, 6 (2014), 2122–2127.
Limpaecher, A, Feltman, N, Treuille, A, and Cohen, M. (2013). Real-time Drawing Assistance Through Crowdsourcing. ACM Trans.
Graph. 32, 4, Article 54 (2013), 8 pages. DOI:http://dx.doi.org/10.1145/2461912.2462016
Lin, C. H, Weld, D. S, and others, . (2012). Dynamically switching between synergistic workflows for crowdsourcing. In Workshops
at the Twenty-Sixth AAAI Conference on Artificial Intelligence.
Little, G, Chilton, L. B, Goldman, M, and Miller, R. C. (2010). TurKit: Human Computation Algorithms on Mechanical Turk. In
Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST ’10). ACM, New York, NY,
USA, 57–66. DOI:http://dx.doi.org/10.1145/1866029.1866040
Liu, Q, Peng, J, and Ihler, A. (2012). Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems.
–700.
Loparev, A, Lasecki, W. S, Murray, K. I, and Bigham, J. P. (2013). Introducing Shared Character Control to Existing Video Games.
(2013).
Mao, A, Chen, Y, Gajos, K. Z, Parkes, D, Procaccia, A. D, and Zhang, H. (2012). TurkServer: Enabling Synchronous and Longitudinal
Online Experiments. In Fourth Workshop on Human Computation (HCOMP 2012).
Mao, A, Kamar, E, and Horvitz, E. (2013). Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing. In First AAAI
Conference on Human Computation and Crowdsourcing.
Mashhadi, A. J and Capra, L. (2011). Quality control for real-time ubiquitous crowdsourcing. In Proceedings of the 2nd international
workshop on Ubiquitous crowdsouring. ACM, 5–8.
Mason, W and Suri, S. (2012). Conducting behavioral research on Amazonâ˘A ´ Zs Mechanical Turk. Behavior research methods 44, 1
(2012), 1–23.
Messias, J. V, Spaan, M, and Lima, P. U. (2011). Efficient offline communication policies for factored multiagent POMDPs. In
Advances in Neural Information Processing Systems. 1917–1925.
Moldovanu, B and Sela, A. (2001). The optimal allocation of prizes in contests. American Economic Review (2001), 542–558.
Moldovanu, B and Sela, A. (2006). Contest architecture. Journal of Economic Theory 126, 1 (2006), 70–96.
Osentoski, S, Crick, C, Jay, G, and Jenkins, O. C. Crowdsourcing for closed loop control. (????).
Parkes, D. C and Procaccia, A. D. (2013). Dynamic Social Choice with Evolving Preferences. (2013), 767–773.
Proper, S and Tadepalli, P. (2009). Solving multiagent assignment markov decision processes. In Proceedings of The 8th International
Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and
Multiagent Systems, 681–688.
Quinn, A. J and Bederson, B. B. (2011). Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 1403–1412. DOI:
http://dx.doi.org/10.1145/1978942.1979148
Raykar, V. C, Yu, S, Zhao, L. H, Valadez, G. H, Florin, C, Bogoni, L, and Moy, L. (2010). Learning from crowds. The Journal of
Machine Learning Research 11 (2010), 1297–1322.
Retelny, D, Robaszkiewicz, S, To, A, Lasecki, W, Patel, J, Rahmati, N, Doshi, T, Valentine, M, and Bernstein, M. (2014). Expert
Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology
(UIST ’14). ACM, New York, NY, USA, 12.
Rossen, B and Lok, B. (2012). A crowdsourcing method to develop virtual human conversational agents. International Journal of
Human-Computer Studies 70, 4 (2012), 301 – 319. DOI:http://dx.doi.org/10.1016/j.ijhcs.2011.11.004
Russell, B. C, Torralba, A, Murphy, K. P, and Freeman,W. T. (2008). LabelMe: A Database andWeb-Based Tool for Image Annotation.
Int. J. Comput. Vision 77, 1-3 (May 2008), 157–173. DOI:http://dx.doi.org/10.1007/s11263-007-0090-8
Sheng, V. S, Provost, F, and Ipeirotis, P. G. (2008). Get another label? improving data quality and data mining using multiple, noisy
labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM,
–622.
Singh, P, Lasecki, W. S, Barelli, P, and Bigham, J. P. (2012). HiveMind: A Framework for Optimizing Open-Ended Responses From
the Crowd. Technical Report. URCS Technical Report.
Snow, R, O’Connor, B, Jurafsky, D, and Ng, A. Y. (2008). Cheap and Fast—but is It Good?: Evaluating Non-expert Annotations for
Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP ’08).
Association for Computational Linguistics, Stroudsburg, PA, USA, 254–263. http://dl.acm.org/citation.cfm?id=1613715.1613751
Spaan, M. T and Oliehoek, F. A. Tree-Based Solution Methods for Multiagent POMDPs with Delayed Communication. In Proc. of
th Benelux Conference on Artificial Intelligenceg:
Surowiecki, J. (2005). The wisdom of crowds. Random House LLC.
Van Kleek, M. G, Bernstein, M, Panovich, K, Vargas, G. G, Karger, D. R, and Schraefel, M. (2009). Note to Self: Examining
Personal Information Keeping in a Lightweight Note-taking Tool. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’09). ACM, New York, NY, USA, 1477–1480. DOI:http://dx.doi.org/10.1145/1518701.1518924
von Ahn, L. (2005). Human Computation. Ph.D. Dissertation. Carnegie Mellon University, Pittsburgh, PA.
von Ahn, L and Dabbish, L. (2004). Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI ’04). ACM, New York, NY, USA, 319–326. DOI:http://dx.doi.org/10.1145/985692.985733
Zhang, H, Law, E, Miller, R, Gajos, K, Parkes, D, and Horvitz, E. (2012). Human Computation Tasks with Global Constraints. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 217–226.
DOI:http://dx.doi.org/10.1145/2207676.2207708
Zhou, D, Basu, S, Mao, Y, and Platt, J. C. (2012). Learning from the wisdom of crowds by minimax entropy. In Advances in Neural
Information Processing Systems. 2195–2203.
Downloads
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
