The Automated State: How 20th-Century Thinkers Foresaw Our AI-Driven Present

One of the stranger things about this dizzying, headlong moment in time is that it doesn’t have much of a past. Everything is about the future of this, the future of that: the future of work, the future of humanity, the future of the planet. It’s as if everyone is screaming (some ecstatically, most terror-stricken): robots are taking over the world! Meanwhile, you can’t put down your phone, unplug, delete your AI apps, tell Zoom to piss off; it feels as if you are racing toward something, and can’t stop, or look back, or think straight.

But of course this weird moment in history does have a past. It comes from somewhere. It wasn’t inevitable. Things could have turned out differently. They still can. And if you look back, one thing you’ll see is that a lot of people saw this all coming. They foresaw our present when it was still the future. They thought it might turn out fabulously, a computer utopia. And they worried it might turn out disastrously. Above all, they issued warning after warning about the rise of what I call the artificial state, the replacement of the liberal democratic nation-state, the consent of the governed, with the rule of people by machines. They warned of a future in which democratic deliberation would be replaced by prediction by calculation, the public sphere would be supplanted by data-driven commerce, and drones would take the place of the demos. Why didn’t anyone listen?

Between the 1950s and the 1980s, computers got smaller, faster, cheaper and more powerful; computer languages became more sophisticated; and electronic data, once a molehill, became a mountain. As data processing and general-purpose computers moved from the military to industry, business and medicine, sizeable chunks of public life became automated, driven by computers, calculations, networks and algorithms that did the political work of rallying support, running campaigns, communicating with constituents and crafting policy, especially in the world’s wealthiest nations.

The adoption and diffusion of such technologies led to wild-eyed speculation about their possible social and political consequences, a discussion that, characteristically, ricocheted between dreams of a technological utopia and fears of an artificial state.

In 1981, for instance, the Japanese sociologist Yoneji Masuda, a theorist of what he called “information society”, predicted the arrival of either a computopia that would bring about the eradication of all distinctions of class and nation and “a symbiosis in which man and nature can live together in harmony”, or a compudystopia that he called an “automated state”: government by machine. Yet the two possible futures had much in common, since the path toward either had already led humans “to neglect the need for coexistence with nature, while our impact on nature has grown immeasurably”.

A turning point in the story of the automated state had come in 1958, when a team of American researchers and ad men built a machine that they claimed could both predict and influence voting behaviour. That year, a landmark conference on the “mechanisation of thought processes” took place at the National Physical Laboratory in Teddington, near London. Speakers included French, Israeli, Russian and Hungarian scientists, and they talked of a future in which computers would one day compose music, resolve legal disputes, control air traffic, conduct surgery and, in the administration of government, replace not only clerks but executives.

In New York, Edward L Greenfield, the head of a Madison Avenue advertising firm, drafted a proposal for what would become known as a People Machine. Greenfield proposed to compile hundreds of thousands of punch cards – election returns and public opinion surveys and census data – to build an “information bank” that could sort voters by types and their opinions by issues. “There is nothing mysterious about this on the surface, the input to the machine being the information about real individuals obtained in surveys”, Greenfield and the MIT political scientist Ithiel de Sola Pool wrote in 1958. But “once this information is inside the high speed storage facilities of the machine, it is a different world”. The machine would operate as a “macroscope” that could simulate the whole of the US electorate.

The next year, Greenfield and Pool and IBM’s Alex Bernstein formed a company called the Simulmatics Corporation. By “simulmatics”, they meant the automated simulation of human intelligence. They believed they had invented “the A-bomb of the social sciences”.

Their first client was the Democratic National Committee. In 1959, the DNC hired Simulmatics to run simulations on an artificial electorate and tell the party’s nominee what to say, to whom, and when. Not everyone was enthusiastic. “I shudder at the implication for public leadership of the notion … that a man shouldn’t say something until it is cleared with the machine,” wrote the historian Arthur Schlesinger.

After John F Kennedy won the nomination, his campaign hired Simulmatics to run predictions on an IBM 704. When Kennedy won, many in the press attributed the victory to Simulmatics’ People Machine. As the New York Herald Tribune put it, “a big, bulky monster called a ‘Simulmatics’” had been Kennedy’s “secret weapon”.

Simulmatics went on to new campaigns, heralded by the press. But the People Machine turned out to have been ahead of its time. In 1970 Simulmatics went bankrupt, unable to find enough paying clients for its services. The problem wasn’t the computers, or the model. The problem was an insufficiency of data. When Bernstein met with book and magazine publishers and film distributors in hopes of selling a market simulation of book sales, magazine subscribers and ticket sales – the kind of microtargeting done decades later by companies such as Amazon and Netflix – he reported that there was no way to build a model for any of these industries.

Simulmatics’ only real successes were in politics, where the company could rely on large caches of data: censuses, election returns and public opinion surveys. But in this sector, the company’s work raised considerable alarm. It also inspired prophecies of political doom, including two dystopian novels. One was published in 1964: The 480, a political thriller in which a barely disguised “Simulations Enterprises” meddles with a US presidential election. It was written by Eugene Burdick, who’d been asked to join the company but had declined, for reasons that became clear in The 480, in which he described Simulmatics as part of a sinister political landscape.

The new underworld is made up of innocent and well-intentioned people who work with slide rules and calculating machines and computers which can retain an almost infinite number of bits of information as well as sort, categorise, and reproduce this information at the press of a button. Most of these people are highly educated, many of them are PhDs, and none that I have met have malignant political designs on the American public. They may, however, radically reconstruct the American political system, build a new politics, and even modify revered and venerable American institutions – facts of which they are blissfully innocent.

Burdick died in 1965, but his concerns about an automated state were bubbling up all over the place. In 1966, in The Myth of the Machine, the American critic Lewis Mumford lamented the rise of “cybernetic intelligence”, warning that under its “automatic operation … man will become a passive, purposeless, machine-conditioned animal whose proper functions, as technicians now interpret man’s role, will either be fed into the machine or strictly limited and controlled for the benefit of depersonalised, collective organisations”.

Mumford described technological determinism – the idea that machines determine the course of history and represent a march of progress – as “a radical misinterpretation of the whole course of human development”. This mistaken belief, he wrote in The New Yorker the next year, had to be abandoned “if we are to get an adequate grip on our mechanised culture before we lose both our consciousness of human purpose and our confidence in being able to control our own creations”.

No one really ever paused to get that grip. Instead, data collection became crucial to industries that began accumulating, as stored computer files, everything from credit card transactions to car rentals to library checkout records, which made possible the kinds of predictions that Simulmatics Corp had only been able to dream of. With so much data, computers – increasingly linked together – stood poised to become far more than simple question-and-answer devices.

While all this was going on – the accumulation of data, refinements in computer architecture and the use of computers not only to calculate but also to communicate – the best thinkers of the age wondered what it might mean for humanity: computopia, or the automated state? The answer appeared to depend, in large part, on what these technologies might mean for humans’ capacity to understand the world. As ever, the nature of government would depend on the nature of knowledge. “Organised knowledge puts an immense amount of power in the hands of people who take the trouble to master it,” as a director of medicine and science at the Rockefeller Foundation put it.

Meanwhile, the nature and form of knowledge was changing. In 1965, the brilliant engineer JCR Licklider, the man most appropriately credited with inventing the internet, wrote Libraries of the Future, in which he considered the many disadvantages of books. “Surveying a million books on 10,000 shelves,” he said, is a nightmare. “When information is stored in books, there is no practical way to transfer the information from the store to the user without physically moving the book or the reader or both.” But convert books into data that can be read by a computer and you can move data from storage to the user, and to any number of users, much more easily.

Using the contents of all the books held in the Library of Congress as a proxy for the sum total of human knowledge, Licklider calculated its size in 1960 and estimated that it was doubling every year. Using these numbers, the sum total of human knowledge, as data, would be about a dozen petabytes in the year 2020. No one really knows how big the internet was in 2020, but some people said it was dozens of zettabytes. If they were right, Licklider was way off. In any event, the library of human knowledge seemed, suddenly, breathtakingly vast.

Would the libraries of the future be available to everyone? And what about the libraries of the present: the troves of data in the possession of governments? In the US, a proposal to establish a National Data Center, meant to provide, for electronic data, what the Library of Congress provided for books and the National Archives for manuscripts, was abandoned due to privacy concerns. (Critics dubbed the proposed National Data Center the “snooping machine”.)

Licklider proved an excellent futurist. In 1963, he’d sent a memo to colleagues at the Advanced Research Projects Agency (Arpa), proposing “to develop a capability for integrated network operation”, what would become first Arpanet and, eventually, the internet.

Licklider’s vantage on this project, and his involvement in most of the cutting-edge research done in the 1960s, gave him exceptional clarity about the future. In April 1968, in an essay for the magazine Science and Technology, Licklider and the psychologist Robert W Taylor predicted that “in a few years, men will be able to communicate more effectively through a machine than face to face”. They also predicted the emergence of “online interactive communities” that would change how people engage with each other: “Life will be happier for the online individual because the people with whom one interacts most strongly will be selected more by commonality of interests and goals than by accidents of proximity.”

Simulmatics Corporation’s co-founder Ithiel de Sola Pool also contributed an essay to that issue, in which he predicted that developments such as personalised newspapers would lead to an era of hyperindividualism. “A reader starts by asking for a digest of the news,” Pool imagined. “He can ask for details on any story that interests him.”

Pool believed the hyperpersonalisation of news would transform politics. “In the 21st century, the sort of critic who now attacks conformity in society may be complaining of an atomised society,” Pool predicted. “Modern technology, he’ll assert, has destroyed our common cultural base and has left us living in a little world of his own.”

He was not wrong. “In the coming atomised society, the information the citizen gets will arise from his own specific concerns,” he wrote, and “when everyone can select his own fund of information, the political problem of gathering an effective body of support behind a particular rational issue or candidate will become very different and very much more difficult”.

More common in the 1960s and 70s than Pool’s predictions of an atomised society under an automated state – or at least louder, and more attention-getting – were predictions of computer-driven socialist revolution or democratic liberation as types of computopias. In Chile between 1971 and 1973, during the rule of the democratically elected Salvador Allende, cyberneticists developed Project Cybersyn, a network whose chief purpose was to manage Chile’s socialist economy. Political reality intervened: the entire project was abandoned after Augusto Pinochet took power in a coup in 1973.

Elsewhere, optimism only grew. “Ready or not, computers are coming to the people,” Stewart Brand predicted in Rolling Stone in December 1972. The personal computer, Brand predicted, would bring “power to the people”. Brand, an LSD advocate, had studied at Stanford, and he described the arrival of Arpanet as the best news “since psychedelics”.

Computopianism found many adherents. Writing in 1975, the Israeli-American sociologist Amitai Etzioni and his co-authors predicted that the computer revolution would bring a return to “the form of democracy found in the ancient Greek city-state, the kibbutz and the New England town meeting, which gave every citizen the opportunity to directly participate in the political process”. Historian Daniel Boorstin, the librarian of Congress, predicted in 1978 that the new ties of communication would create new ties of community, a development he called the “republic of technology”.

Yet there were, as ever, sceptics. Joseph Weizenbaum, an MIT engineer who in 1966 had developed Eliza, the first of what would come to be called a chatbot, objected to the development of artificial intelligence as leading, inevitably, to the automation of the functions of the state. Writing in 1976, he described as “obscene” any project that proposed “to substitute a computer systems for some aspect of human interaction that involves interpersonal respect, trust, and mutual understanding.”