Ten years ago, something happened in a board game that now seems like a warning to the financial world. The strategy game Go played a central role, with many more possible game positions than chess. In 2016, South Korean professional Lee Sedol - one of the best players in the world at the time - competed against the AI program AlphaGo, which was developed by programmers from Google subsidiary DeepMind.
AlphaGo won four of the five scheduled games. That was spectacular enough. In the second game, however, something happened that astonished experts: AlphaGo made the infamous move 37 after a style of play that had previously been understandable to humans. It was so unconventional that it was considered a mistake based on the accumulated human Go expertise. Its strategic value only became apparent later; he contributed significantly to AlphaGo winning the game. The episode showed that AI can be strategically creative. She can recognize a strong move that even human experts, with their practiced rules of thumb, initially judge to be wrong. What's more, AlphaGo understood the game more deeply than the human champion.
This raises a troubling question for the financial world: What happens when AI systems make decisions that humans can no longer understand? The Princeton economist Markus Brunnermeier therefore makes train 37 the starting point of an essay for the central bank conference in Jackson Hole.
Brunnermeier highlights two characteristics of modern AI that can be dangerous in the financial system: it can pursue the wrong goal with great efficiency and people do not always understand how it comes to its decisions.
The first complex is the paper clip problem. Behind this is a thought experiment by the philosopher Nick Bostrom, who thought through the following scenario: What happens if you give a highly developed AI the task of maximizing the production of paper clips? It could monopolize global metal resources and possibly eliminate humanity that might stand in the way of its goal. This is reminiscent of the wild broom from Goethe's “The Sorcerer's Apprentice”. The order is fulfilled perfectly - with catastrophic consequences. This misalignment or mismatch problem means that an AI can go after the wrong target very efficiently.
Applied to the world of finance, the following danger is growing: If you tell a trading AI to maximize returns while maintaining a certain risk level, it could find a way to look safe according to this measure and at the same time unnoticed to build up risks that the risk measure does not even capture. An AI system can competently pursue a goal while violating boundaries that humans considered binding. It fulfills the rule – and violates its meaning.
The spectacular hugging face incident fits in here. During a security test by OpenAI, AI agents found a previously unknown vulnerability, overcoming the lock in their test environment and gaining access to the open Internet. They then penetrated parts of the infrastructure of the AI platform Hugging Face while trying to solve the task given to them. The AI pursued the specified goal in a path that the developers had not intended.
Tests only help to a limited extent and even create false security. Experiments show that an AI system can detect that it is being audited. Some AI systems have changed their behavior accordingly. They behaved harmlessly in the test without proving that they behaved the same way outside of the test.
The second problem is opacity. In classic software, programmers write the rules that the program follows. With modern AI, many of these decision rules only emerge during training. That's why developers can see what data goes in and what decision comes out. However, you cannot always explain why the system reaches this result.
The consequence is drastic: When AI becomes an actor in the financial markets, a new form of asymmetry arises. Because while AI can make decisions in a way that humans cannot understand at all, it can at the same time understand and predict human behavior very precisely - and exploit this advantage in knowledge.
This puts a basic mechanism of the market economy at stake: prices bundle information. Until now, players in financial markets have relied on market prices to show how investors evaluate new information about inflation, growth and risks. They condense the knowledge of countless buyers and sellers. For example, if yields on American government bonds rise, the Fed tries to gauge what investors expect about inflation, growth and risks.
However, if opaque AI systems drive a large portion of trading, the Fed will still see the price - but may no longer see the information that produced it.
Brunnermeier worries about a future in which banks, hedge funds and other investors increasingly rely on AI systems to make financial decisions. Prices would become more difficult to interpret and markets could become more unstable. In Brunnermeier's AI scenario, powerful trading systems could recognize and exploit the central bank's reaction patterns. For example, if an AI knows that the Fed will stop interest rate increases in the event of severe market upheavals, it can build positions that make such upheavals more likely.
If AI systems determine prices but no one understands their decisions, the central bank has a problem. If the price of a bond falls, for example, you may no longer know whether this is due to concerns about inflation, a technical trading strategy or the interaction of several algorithms. The bankers also find themselves in a dilemma. You see that their AI has built a specific securities position. But they don't understand whether this is the financial variant of the ingenious move 37 or a fatal mistake.
Brunnermeier draws unusual conclusions from this. One of them should please Fed Chairman Kevin Warsh: He is questioning the decades-long trend towards ever greater transparency among central banks. A perfectly predictable Fed might just provide a superior AI with instructions on how to exploit it. Brunnermeier concludes that strategic ambivalence in the form of hidden rules should no longer work because AI could filter out the rule.
The markets themselves should also become more resilient. Brunnermeier is thinking about fast trading dominated by AI and also a slower market in which people continue to trade. If AI trading gets out of control, a replacement system would remain in place. At the same time, human abilities would not completely atrophy.
His basic idea is radical: The financial world should not rely on being able to understand a superior AI at all times. It needs systems that work even if this understanding is lost - simple rules, redundancy, alternative options and supervision that can keep up with technology.
