Study says AI data taints vital human input

Earlier this century, Jeff Bezos popularized the use of low-paid Turkish mechanical workers who work remotely with perhaps thousands of others on tiny parts of larger computer projects to ensure a human perspective on mostly simple tasks which have proven to be perplexing for computers. He called this fusion of human and digital brain power “artificial artificial intelligence.”

About a quarter of a million people are employed through Amazon’s Mechanical Turk marketplace, just one of many sources providing such services.

This week, researchers at the Swiss university EPFL reported that Turks who had made an important human contribution are now relying on AI-generated content to complete their tasks. They dubbed this phenomenon “artificial artificial intelligence”.

The term may elicit smiles, but researchers say the findings raise serious concerns.

Workers tapping into AI generators to perform their tasks “would greatly reduce the usefulness of the crowdsourced data,” said researcher Veniamin Veselovsky. The paper, “Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks”, was published in the arXiv prepress server June 13th.

While large language models excel at processing training data, human input is still superior for certain tasks. Humans tag data entered into templates, describe images, and respond to CAPTCHA screens more efficiently than computers.

“It’s tempting to rely on crowdsourcing to validate the results of a large language model or to create human-standard data for comparison,” Veselovsky said. “But what if crowdworkers themselves use LLMs to increase their productivity, and therefore their income, on crowdsourcing platforms?”

Such inputs would taint the data pool and, if not addressed, could bring into question the reliability of AI-powered operations.

The term “Turk” comes from an 18th-century chess master “robot” who defeated players all over Europe. Napoleon and Benjamin Franklin were among the defeated. The unsuspecting players never knew that a human chess expert was hidden under the boards of the machine.

Crowdsourcing with modern Turks has become a billion dollar industry. Its reputation has been tarnished by the notoriously low wages some companies pay their workers. Turks make $2 to $5 an hour.

But the industry is threatened by the sudden adoption of large language models. According to a recent study, a ChatGPT 3.5 turbo model tackling grading assignments performs significantly better than crowd workers at about one-twentieth the cost.

Workers will face increased pressure to produce more and do it faster, and this in turn could lead those workers to rely more on AI resources.

Based on a limited study of the use of large language templates by workers at MTurk, Amazon’s crowdsourcing operation, EPFL researchers estimated that 33% to 46% of worker assignments were completed with the help of large language models.

“Large language models are becoming more popular by the day, and multimodal models, which support not only text but also image and video input and output, are on the rise,” said Veselovsky . “With this, our findings should be regarded as the ‘canary in the coal mine’ which should remind platforms, researchers and crowd workers to find new ways to ensure human data remains human.”