Making crowdworking more efficient, ethical, fair and trustworthy

Crowdsourcing platforms have created a rapidly growing online labor market, particularly for jobs that need human intelligence or are too complex to be automated.

Job requesters such as companies and researchers post tasks – say, answering a survey or labeling a set of data to train an artificial intelligence program – then workers from around the world complete them and money changes hands. Or at least that’s how it’s supposed to work.

“There are a number of critical problems around fairness, ethics, efficiency and trustworthiness,” said Chuan Yue, associate professor of computer science at Colorado School of Mines. “They significantly impact the healthy development of this online labor market and what can be achieved by using these platforms.”

Yue is the lead researcher on a $1 million National Science Foundation project that aims to address those challenges and make crowdsourcing more efficient, ethical, fair, and trustworthy for workers and job requesters alike.

An expert in cybersecurity and privacy, Yue will work with two Mines colleagues on the interdisciplinary project – Qin Zhu, assistant professor of humanities, arts & social sciences and an expert in science and engineering ethics, and applied microeconomist Ben Gilbert, assistant professor of economics and business.

Together, their goal is to develop incentive structures based on economic theory to encourage fairness; training and assessment mechanisms to incorporate ethics; and machine learning models to improve efficiency and securely protect both crowdworkers and job requesters.

“Over the years, there has been so much fear that artificial intelligence may replace humans and eliminate a large number of traditional jobs in the near future. Many traditional jobs will be replaced, for sure, but AI still needs significant collaboration from humans,” Yue said. “Specifically, there is a significant need for humans to help label ground-truth data to help facilitate the wide adoption and proper application of AI techniques – that labeling cannot be replaced now or in the near future, but in order for these techniques to reach their full potential, we need a healthier online job market.”

A recent study showed that crowd workers earn $2-$3 per hour on average – assuming they are paid at all, Yue said.

“There’s no accountability for mistreating these online workers,” he said. “Typically, their payment is pretty low – lower than federal regulations – and they may not receive pay at all, since requestors have the authority to reject the payment and they don’t have to provide a reason.”

Both workers and requestors can be vulnerable to malicious intent as well, whether that’s requestors violating workers’ privacy by collecting sensitive information or taking them to malicious websites, or workers manipulating the task results by using malicious programs.

Researchers have interviewed more than 30 current crowdworkers to inform their work.

“We are going to listen to the needs of crowdworkers and incorporate their social and ethical concerns into the reimagining and redesigning of crowdsourcing platforms,” Zhu said. “We are hoping our project can help future American workers better prepare for the employment challenges brought by artificial intelligence.”

Funding for the project comes from NSF’s new Convergence Accelerator Pilot program, which aims to fast-track “use-inspired, convergence research in areas of national importance.” (Another project at Mines, led by Mechanical Engineering’s Xiaoli Zhang, to create an AI-enabled tool to develop faster, more individualized retraining for displaced workers, also won an accelerator grant.)

“There are so many existing tasks that could be dispatched through these platforms – I think we’ve barely scratched the surface on the impact this will have on the broader labor market. That’s why it’s so important to get this right,” Gilbert said. “Workers benefit from the flexibility, but you also have all the problems of an unregulated labor market and none of the benefits of a stable employer, combined with a host of new cybersecurity challenges that arise from operating that market online. There are huge potential gains from trade, but there is also huge potential for abuse.”