Recently, beyond rapid Development of large language models (LLMs), significant progress has been made in AI game theory as well. After completing Pluribus, Noam developed Cicero by integrating game-theoretic reinforcement learning with LLM technology. Cicero learned to employ humor and form alliances in a diplomatic strategy game, achieving top 10% performance, with opponents completely unaware they were playing against an AI. Noam has recently moved to OpenAI, likely to develop something referred to as "Q*". It is believed that combining reinforcement learning with game theory and large language models will have practical applications in gaming, autonomous driving, and robotics planning. Here, I will share insights on the principles and technical challenges of Pluribus, particularly within the context of Texas Hold'em poker.
I plan to write seven or eight chapters, using concepts from Texas Hold'em to help explain Nash Equilibrium, a concept commonly associated with the prisoner's dilemma. Nash Equilibrium is a state where no player can increase their payoff by unilaterally changing strategies, assuming other players' strategies remain unchanged. While Nash Equilibrium does not guarantee maximizing collective or individual benefits, it provides a non-exploitative strategy when opponents' strategies are unknown. By analyzing deviations from Nash Equilibrium, one can infer opponents' habits and devise strategies to maximize overall benefits.
In the game of rock-paper-scissors, if you use a Nash equilibrium strategy where the probabilities of choosing rock, paper, and scissors are all 33%, your opponent will not be able to exploit any weaknesses. However, if you choose rock 100% of the time, your opponent can exploit you by choosing paper 100% of the time.
For instance, in a Texas Hold'em game at the river stage, suppose the community cards include four twos and a heart three, and the current pot is 2. A leading player with a 50% chance of holding AA and a 50% chance of holding QQ can choose to check or bet 1. If this player bets and the opponent holds KK, the opponent might decide to fold or call.
Using Nash Equilibrium solvers, such as the one provided by bupticybee on their GitHub (https://github.com/bupticybee/TexasHoldemSolverJava), we find:
For Player 1:
With AA, bet 1 100% of the time.
With QQ, check 66% and bet 1 33% of the time.
For Player 2, facing a bet:
With KK, call 66% and fold 33% of the time.
These strategies ensure that neither player can improve their expected payoff by deviating unilaterally from this strategy set.
First, let's discuss Player 1's actions. The expected value of the Nash equilibrium strategy is:
0.5×0.66×2 (AA bets 1, opponent calls and wins 2)+
0.5×0.33 (AA bets 1, opponent folds and wins 1)+
0.5×0.33×0.66×(−2) (QQ bets 1, opponent calls and loses 2)+
0.5×0.33×0.33 (QQ bets 1, opponent folds and wins 1)+
0.5×0.66×(−1) (QQ checks, opponent bets, folds and loses 1)=0.33
If Player 2 adjusts their strategy to 50% calling and 50% folding in response to the Nash equilibrium strategy, Player 1's betting expected value becomes 0.66.
If Player 2 adjusts their strategy to 100% calling in response to the Nash equilibrium strategy, Player 1's betting expected value becomes 0.33.
Player 2 cannot reduce Player 1's expected value by modifying their strategy; since this is a zero-sum game, Player 2 cannot increase their own expected value either. Note that the calculation for Player 1's check line is omitted for simplicity here.
If Player 1 modifies their strategy to always bet 1 with AA and check with QQ, then Player 2 can adjust their strategy to fold 100% against a bet, making Player 1’s betting expected value 0. This situation in poker is known as "under-bluffing."
If Player 1 modifies their strategy to always bet 1 with both AA and QQ, then Player 2 can respond by calling 100%, reducing Player 1’s expected value to 0. This situation in poker is known as "over-bluffing."
If Player 1 deviates from the Nash equilibrium strategy, Player 2 can adjust their strategy to reduce Player 1's expected value.
Some clever friends have pointed out that using Nash equilibrium strategies is a defensive approach that cannot yield profits. In practice, however, opponents often self-exploit unintentionally. For example, the expected value of a 50% bet and 50% check strategy is 2, while the opponent’s choice of 30% check and 70% bet has an expected value of 1, effectively causing them to exploit themselves.
The real Texas Hold'em game tree has 10^180 nodes, and real matches require decisions within 30 seconds. This has led to the development of various sophisticated CFR (Counterfactual Regret Minimization) methods and numerous engineering optimization tricks, which I will gradually introduce later.
In the next chapter, I will discuss algorithms related to CFR and virtual regret minimization.
I am looking for remote job opportunities. Please contact me at my email: justfunnychen@gmail.com.
Top comments (0)