Time: 2 hours
Time span: Dec 19 – Dec 20, 2 days
Panagiotis G. Ipeirotis, Raman Chandrasekar, and Paul Bennett. 2010. A report on the human computation workshop (HComp 2009). SIGKDD Explor. Newsl. 11, 2 (May 2010), 80-83.
Panagiotis G. Ipeirotis(个人主页, 博客, publication), 2004年从哥伦比亚大学获得博士学位，现在是Leonard N. Stern School of Business of New York University的associate professor, 研究方向为"crowdsourcing and on mining user-generated content on the Internet"， 在好几个牛会获得过Best Paper(IEEE ICDE 2005, ACM SIGMOD 2006, WWW 2011)和Best Paper Runner Up awards(JCDL 2002, ACM KDD 2008), 另一位神人Nikolay Archak正是他的学生(上个月刚通过博士答辩)。Ipeirotis2004年获得博士学位，仅2年之后就能带博士生Archak, 可见国外体制很灵活。Ipeirotis门下有弟子四人，其中两位是大陆过去的女博。
这篇文章是关于第一届human computation workshop的一份报告（Ipeirotis是这个workshop的Organizing Committee），主要就是介绍了这个workshop的papers、demos、posters，以及总体的情况(HComp2009@dblp)。
1. 一些数据
- 这个workshop共收到33份提交
- 录用率是1/3
- workshop开了半天，有40人参加
- workshop开场白是Luis von Ahn做invited talk
- 评出了两篇best paper:
- Financial Incentives and the Performance of Crowds, by Winter Mason and Duncan J Watts.
- KissKissBan: A Competitive Human Computation Game for Image Annotation, by Chien-Ju Ho, Tao-Hsuan Chang , Jong-Chuan Lee, Jane Yung-Jen Hsu and Kuan-Ta Chen.
2. 此次workshop有4个session，依次为:
- game
- Luke Barrington, Damien O'Malley, Douglas Turnbull, Gert R. G. Lanckriet: User-centered design of a social game to tag music. KDD Workshop on Human Computation 2009: 7-10
use an active learning approach to tag music
- Chien-Ju Ho, Tao-Hsuan Chang, Jong-Chuan Lee, Jane Yung-jen Hsu, Kuan-Ta Chen: KissKissBan: a competitive human computation game for image annotation. KDD Workshop on Human Computation 2009: 11-14
基于ESP的一个游戏，解决两个问题：（1）如何应付作弊？（2）选手提供的tag太有限。 需要先读ESP的论文后再来看这篇。
- Yen-ling Kuo, Jong-Chuan Lee, Kai-yang Chiang, Rex Wang, Edward Shen, Cheng-wei Chan, Jane Yung-jen Hsu: Community-based game design: experiments on social games for commonsense data collection. KDD Workshop on Human Computation 2009: 15-22
- demos & posters
- 包括以下8个Demos
- The Phrase Detectives Game, from the University of Essex, to help create large-scale linguistically annotated corpora. http://anawiki.essex.ac.uk/
- Picture This, a game from Microsoft to elicit preference judgments to rank images for an image query. http://www.clubbing.com/Pages/Games/GameList.aspx?game=Picture_This
- Page Hunt, also from Microsoft, to garner data to improve web search relevance, going from web pages to queries. http://pagehunt.msrlivelabs.com/PlayPageHunt.aspx
- TurkIt, from MIT, providing a library and toolkit for iterative tasks? on Mechanical Turk.
- Seaweed, also from MIT, to help design economic games.
- Search War, from CMU, to collect, for a given web page, its relevance and salience.
- Magic Bullet, from Newcastle University, to streamline the robustness evaluation of CAPTCHAs.
- Thumbs-UP, from Yahoo!, to rank search results.
- Games for Games, from Est Creativity Rising and the University of Bremen, examining how human computation game design and scoring approaches affect the quality of data gathered.
- 包括以下5个posters
- From active towards InterActive learning: using consideration information to improve labeling correctness. 40-43
- TagCaptcha: annotating images with CAPTCHAs. 44-45
- CAPTCHA-based image labeling on the Soylent Grid. 46-49
- Designing crowdsourcing community for the enterprise. 50-53
- A reputation system for selling human computation. 54-57
- game theory
- Shaili Jain, David C. Parkes: The role of game theory in human computation systems. KDD Workshop on Human Computation 2009: 58-61
basic idea: human computation may benefit by using game theoretic concepts to improve the design of the games.
- Chien-Ju Ho, Kuan-Ta Chen: On formal models for social verification. KDD Workshop on Human Computation 2009: 62-69
describe how to use game theory to show the effect of sequential verification vs. parallel verification.
- labeling cost & eficiency
- Ran Gilad-Bachrach, Aharon Bar-Hillel, Liat Ein-Dor: Efficient human computation: the distributed labeling problem. KDD Workshop on Human Computation 2009: 70-76
要解决的问题：label含义的不一致性，比如同一个东西可以用不同词汇描述，同一个词汇也可能有不同含义。作者提出了图论算法来解决这个问题。
- Winter A. Mason, Duncan J. Watts: Financial incentives and the "performance of crowds". KDD Workshop on Human Computation 2009: 77-85
基于对Mechanical Turk的研究，得到了一些关于经济激励与表现的结论。
3. (S6) Themes一章，作者汇总了workshop讨论时大家提的问题，若阅读相关论文可对比着看。
4. 其他资源
这个workshop有一个wiki空间： http://hcomp2009.wikispaces.com/