In 1770, the Hungarian inventor Wolfgang von Kempelen unveiled the Mechanical Turk, a chess-playing contraption that “consisted of a wooden cabinet behind which was seated a life-size figure of a man, made of carved wood, wearing an ermine-trimmed robe, loose trousers and a turban — the traditional costume of an Oriental sorcerer,” according to journalist Tom Standage. The chess-playing robot was toured around Europe and America, and exhibition matches were staged with such famous opponents as Napoleon Bonaparte. All the while, Kempelen maintained that the automaton operated by its own accord.
To prove there was no trickery, he opened the cabinet before every exhibition and showed spectators the dense tangle of gears, wheels, and levers. But Kempelen had actually created an elaborate illusion, not a robot. Inside was a human chess master who used magnets and levers to operate the Mechanical Turk and hid behind the fake machinery when Kempelen opened the cabinet. In other words, the complex mechanical system that Kempelen showed people was meant to distract their attention from how the automaton really worked: human labor. Kempelen sold the idea of an intelligent machine, but what people witnessed was just human effort disguised by clever engineering.
In the 1730s, a French inventor named Jacques de Vaucanson a copper-pated cyborg called La Canard Digérateur, or the Digesting Duck. It was the size of a living duck, walked like a duck, and quacked like a duck. But its real trick, which amazed and baffled audiences, was that it could shit like a duck. The automaton “ate food out of the exhibitor’s hand, swallowed it, digested it, and excreted it, all before an audience,” as journalist Gaby Wood described it in an article for the Guardian.
Vaucanson claimed that he had built a “chemical laboratory” in the duck’s stomach to decompose the food before expelling it from the mechanical butt. While Vaucanson was an expert engineer — the duck was an intricate piece of machinery — like a good magician he did not reveal how the duck worked. After his death, the secret was uncovered: There was no innovative chemical technology inside the duck, rather two containers, one for the food and one for preloaded excrement. (Strangely, the Digesting Duck and Mechanical Turk were both destroyed by museum fires around the same time in the mid-19th century.)
Kempelen and Vaucanson would fit very well into Silicon Valley today. They could make mysterious machines and wondrous claims to the public about what they could do. Vaucanson literally snuck shit into his technological system and called it innovation. And Kempelen’s Mechanical Turk was a forerunner of today’s systems of artificial intelligence, not because it managed to play a game well, as with IBM’s Deep Blue or Google’s AlphaGo, but because many AI systems are, in large part, also technical illusions designed to fool the public. Whether it’s content moderation for social media or image recognition for police surveillance, claims abound about the effectiveness of AI-powered analytics, when, in reality, the cognitive labor comes from an office building full of (low-waged) workers.
We can call this way of building and presenting such systems — whether analog automatons or digital software — Potemkin AI. There is a long list of services that purport to be powered by sophisticated software, but actually rely on humans acting like robots. Autonomous vehicles use remote-driving and human drivers disguised as seats to hide their Potemkin AI. App developers for email-based services like personalized ads, price comparisons, and automated travel-itinerary planners use humans to read private emails. A service that converted voicemails into text, SpinVox, was accused of using humans and not machines to transcribe audio. Facebook’s much vaunted personal assistant, M, relied on humans — until, that is, it shut down the service this year to focus on other AI projects. The list of Potemkin AI continues to grow with every cycle of VC investment.
There is a long list of services that purport to be powered by sophisticated software, but actually rely on humans acting like robots
The term Potemkin derives from the name of a Russian minister who built fake villages to impress Empress Catherine II and disguise the true state of things. Potemkin tech, then, constructs a façade that not only hides what’s going on but deceives potential customers and the general public alike. Rather than the Wizard of Oz telling us to pay no attention to the man behind the curtain, we have programmers telling us to pay no attention to the humans behind the platform.
When the inner workings of a technology are obscured, it’s often labeled a “black box,” a term derived from engineering diagrams where you can see the inputs and outputs but not what happens in between. An algorithm, for example, might effectively be black-boxed because the technical details are described using dense jargon decipherable by only a small group of experts. Accusations of willful obscurantism are often reserved for postmodernism, but as a recent paper on “troubling trends in machine learning scholarship” points out, research and applications in this field are rife with ambiguous details, shaky claims, and deceptive obfuscation. Being baffled by abstruse critical theory is one thing, but not being able to discern how an AI makes medical diagnoses is much more consequential.
Algorithms might also be black-boxed through the force of law by the tech companies who claim them as trade secrets. In The Black Box Society, Frank Pasquale details how many of the algorithms that govern information and finance —the circulation of data and dollars — are shrouded in opacity. Algorithms are often described as a type of recipes. Just as Coca Cola keeps their formula a tightly guarded secret, so too do tech companies fiercely protect their “secret sauce.” Again, it’s one thing to enjoy a beverage we can’t reverse-engineer, but quite another to take on faith proprietary software that makes sentencing decisions in criminal cases.
Potemkin AI is related to black boxing, but it pushes obfuscation into deception. The Mechanical Turk, like many of the much-discussed AI systems today, was not just a black box that hides its inner workings from prying eyes. After all, Kempelen literally opened his automaton’s cabinet and purported to explain how what looked to be a complex machine worked. Except that he was lying. Similarly, marketing about AI systems deploy technical buzzwords work as though they were a magician’s incantations: Smart! Intelligent! Automated! Cognitive computing! Deep learning! Abracadabra! Alakazam!
Weaving the right spell can endow an AI system with powers of objectivity, neutrality, authority, efficiency, and other desirable attributes and outcomes. Like any good trick, it matters less if the system actually works that way than if people believe it does and act accordingly.
Why go to the trouble of creating Potemkin AI? What’s at stake for those propping up the façade? Broadly, there are two ancient reasons at play: profit and power. If an AI application relies heavily on human labor rather than machine learning, then that doesn’t make for a good sales pitch to venture capitalists and clients nor does it convince the public of your technology’s power. There are, of course, other motivations like fame and recognition, but I think we can safely label them as secondary to profit and power.
Each of the main motivations is illustrated by the following two examples of Potemkin AI: Amazon’s Mechanical Turk and the surveillance systems being deployed in China by the Chinese government. In the former, a dehumanized labor platform provides cheap “intelligence,” while buying time for innovation to finally arrive. In the latter, a dehumanized monitoring apparatus creates the illusion of inescapable control.
Amazon’s Mechanical Turk platform — or, MTurk as it’s called — allows employers to post discrete, often routine tasks like completing surveys or tagging pictures. Workers who complete these micro-jobs are then paid micro-wages: One study calculated the median wage at around $2 an hour. As Leslie Hook noted in an article for the Financial Times, MTurk is sometimes described as “humans-as-a-service,” or the “human cloud,” or even “artificial artificial intelligence” to capture its approach of organizing a legion of human workers — hundreds of thousands of people — scattered across the world and hiding them behind an online platform. Many companies rely on this massive pool of cheap labor ready to click and submit, which allows them to quickly scale up in completing tasks that they hope will one day be accomplished by AI software.
Given that the name Mechanical Turk explicitly references the 18th century hoax, it appears that there is no intention to deceive users about the flesh-and-blood foundations of the system. MTurk is up front about how work is outsourced to real live humans. Whereas Kempelen’s overtly claim that his machine was autonomous, MTurk uses clever design to induce that impression in an audience eager to believe in the platform’s Potemkin trick. As digital labor scholar Lilly Irani describes, MTurk is made to mask the Turkers, essentially dehumanizing the platform. “By rendering the requisition of labor technical and infrastructural,” she writes, MTurk “limits the visibility of workers, rendering them as a tool to be employed by the intentional and expressive hand of the programmer.” The platform and its interfaces allow employers to command people as though they were simply operating a mindless machine. In this case, Potemkin AI provides a convenient way to rationalize exploitation while calling it progress.
The interfaces allow employers to command people as though they were simply operating a mindless machine
In addition to outsourcing menial tasks, Irani explains how Potemkin AI like MTurk has helped compensate both technically and ideologically for the shortcomings of actual AI in completing cognitive tasks “by simulating AI’s promise of computational intelligence with actual people.” Even clickwork that seems brainless and dull is still too advanced for “smart” machines. This simple fact does not bode well for funding of AI research and development, especially when investors eventually expect real results and profitable products. Contrary to their cheery marketing copy, Investors and corporations don’t funnel their money into AI because they are interested in innovation for its own sake. AI promises to solve the problems of capital by unlocking exponential growth, eliminating labor costs, optimizing efficiency, and a slew of other expected outcomes. But the AI solution will come about only if the systems actually eventually work as promised.
There is a looming fear that once reality catches up to the hype another “AI Winter” will arrive, freezing all funding and interest in AI. The first cycle of hype for AI began building in the 1950s and grew until the mid-1970s, when enthusiasm was replaced by disillusionment. The ensuing AI Winter lasted until the late 2000s when the combination of big data, processing power, and digital platforms opened up new advances in machine learning research.
As AI attracted more attention, it became a label that startups could use as a shorthand for calling their service innovative, disruptive, and all around better than their dumb competitors. The inflated claims of what AI could achieve feed an expectation economy sustained by a circular logic: investment leads to promises, which leads to branding and more investment … and so on until the bubble bursts.
Some of the most hyped, most cutting-edge applications of AI are supported by this sort of propaganda. A prime case is the extensive surveillance system being deployed in China. With millions of cameras throughout Chinese cities, the state is looking to upgrade analysis of the feeds, with AI and facial recognition that can automatically identify people and even punish criminals. For example, a camera at a busy intersection can now witness jaywalkers in action, shame them by displaying their information on a screen, and send them a text message with a fine. It is questionable, however, just how automatic this name-and-shame system actually is right now. Buried at the bottom of a recent New York Times article about China’s totalitarian tech is a nugget that highlights how the AI involved in this system is more hype than real:
The system remains more of a digital patchwork than an all-seeing technological network. Many files still aren’t digitized, and others are on mismatched spreadsheets that can’t be easily reconciled. Systems that police hope will someday be powered by AI are currently run by teams of people sorting through photos and data the old-fashioned way … Still, Chinese authorities who are generally mum about security have embarked on a campaign to persuade the country’s people that the high-tech security state is already in place.
Potemkin AI is an effective way of constructing a panopticon. The disciplining power is much greater if people believe that an inhuman force is tirelessly processing feeds from the ubiquitous cameras, rather than groups of human analysts who take time, get fatigued, and make mistakes. Persuading people that the police are using AI is a way to normalize the idea that AI should be and, perhaps more important, already is ceaselessly monitoring society. Again, for the purposes of power and discipline, it matters less if the AI is real or fake — what matters is if people believe in the Potemkin deceit.
It’s easy to say, well, of course the Chinese government would employ propaganda to deceive the public about its power. But it’s simply using a tried and true tactic of Silicon Valley: fake it ‘til you make it. There is a long history of hiding the dead ends and delays in the process of technological development. This makes the process appear to be linear (no divergences), deterministic (no stopping), and progressive (no worries). While, at the same time, it suppresses any skepticism and convinces the public that resistance is futile — a tendency L.M. Sacacas has labeled the “Borg complex” — because the tech is so effective and so much better than any alternatives. You can’t argue with an algorithm, and the AI in the sky is always watching.
To varying degrees, many applications of AI are more like simulations of AI. This isn’t to say that all research and development on artificial intelligence is an elaborate plot to erect a façade of efficacy. Plenty of researchers out there, like my colleague Kazjon Grace, are working to advance the science bit by bit and devise useful applications — such as enhancing creative design and encouraging behavioral diversity — without falling into the traps of obfuscation and overpromising. Yet at the same time, Potemkin AI is not just limited to a few bad actors in an otherwise healthy industry.
The problem isn’t necessarily with AI, per se, or that AI doesn’t work, but rather with the cultural hype and ideological goals that drive the development of AI. Too much of the attention and funding for AI is garnered by those who are looking to maximize their profits and/or secure their power. We see so many attempts to use AI as a tool for replacing human decisions, exploiting human labor, administering human life that it becomes easy to believe these are simply the best, even natural, applications. The AI is an alibi, a way to rationalize these applications. And Potemkin AI is a placeholder, a way to normalize this attitude in advance. But it only works if we don’t look beyond the façade.