Source: https://oliviercouthaud.com/twentyone · Status: Working · 2026 · Tags: Rust, Python, PyTorch, PPO

# TwentyOne

An AI that plays a two-player Roblox card game and can advise me on a move while I play.

## What I wanted

I picked this one because it's a game I've played, and I could see how an AI might get good at it. What I really wanted to know was whether it could beat my friends, my family, or me. My one match against it is described at the end.

Twenty One is a Roblox card game, a take on the 21 minigame in Resident Evil 7. Two players draw cards numbered 1 to 11 toward 21, and whoever is closer without going over wins the round. The loser of each round takes damage: a saw moves toward them by the current bet, which starts at 1 and rises by 1 each round by default, and the match ends when it reaches them. There are also 25 kinds of trump card that change the target, the bet, the totals or the deck, and some of them raise or lower the bet. This is unofficial. I'm not connected to the game's makers, and none of their art or code is in it.

Beyond playing well, I wanted it to be able to advise me in the middle of a round. The rules aren't written down in one place, so the first job was a spec: 45 settings, where every rule I'm not sure about is a flag. A wrong flag makes the advice wrong, so the networks are trained over both readings of the contested rules. The spec starts from the rules as I know them from playing, and I checked them against a community wiki.

## How it works

Figure: Diagram. How the pieces depend on each other. Click, tap or tab to a box (or use the buttons on a phone) for a one-line explanation. The solver only covers the game with trumps switched off.

I implemented the spec twice. One is a Python engine written to be easy to read, at about 10,000 steps a second. The other is in Rust, for speed: around 11 million steps a second for the engine step alone, batched on 24 threads with a random policy (the whole loop with a Python policy runs at 2 to 6 million). A differential fuzzer plays random games on both in lockstep and compares the full state after every step. Its first run covered 210,000 games and 15.6 million steps, and every difference it found was fixed and kept as a regression test.

The solver gives the exact equilibrium of the game with trumps switched off. That is my yardstick for how far a network is from perfect play. I can only compute it without trumps.

**What is an equilibrium here?**

In a game with hidden cards, an equilibrium is a strategy that nobody can beat on average by switching tactics. If both players use it, neither gains anything. So a network's distance from it is how many percentage points of match win rate it gives away to a perfect player. It can be computed exactly only for the simpler game, which is why the yardstick stops at the no-trump version.

The network has about 2.4 million parameters, with a GRU, a small memory layer that carries what it has seen through a match. v1 is recurrent PPO: it plays itself, its own older snapshots and some simple bots.

**What is PPO?**

Proximal Policy Optimization. The network plays, and after each batch of games it is nudged toward the moves that went well and away from the ones that didn't, with a cap on how far one update can push it, so a lucky batch can't wreck it. Here the only reward is winning or losing the match.

v2 adds the solver's answers as extra training targets. v3 is expert iteration: seven cycles of an hour of search labels, half an hour of training and a check, 8.3 hours on my RTX 5070 Ti.

**What is expert iteration?**

A loop between a quick player and a slow one. The slow one, here the search, looks hard at a position and finds a better move than the network's instinct. The network is then trained to produce that move directly, which makes the quick player better and the next round of search better. Training one model to copy a stronger one is called distillation.

Figure: Diagram. One decision in the advisor. The network answers first. The search runs in the background on a copy of the session and replaces the recommendation only when its margin rule says so. Everything is typed in by hand.

The advisor is what I'd use at the table. I type what I see in a short shorthand, a small web app that only runs on my own machine turns that into game events, and the network answers in about 5 ms on the CPU. In the background the search takes a copy of the match, samples hidden-card worlds that fit what I've seen, and plays the rest of the match out with the network, assuming the opponent plays like it. There are three budgets: fast, about 1,000 rollouts and 1 s; normal, 5,000 and 5 s; deep, 16,000 and 30 s. I used to require a lead of two standard errors before the search overrode the network and lowered that to one after measuring what each setting gained per decision. The advisor also warns me when a loss would kill me, or when one of my Bless cards is buried under another trump and isn't protecting me, and it keeps an UNVERIFIED RULES line on every answer until I confirm each unsure rule in the real game. There's a second screen in the same app where I can play against the network at three levels, which is where my one game happened.

### What thinking longer adds

**The same numbers: the judging search plays the opponent as listed. Gain per decision at margin z = 1, with z = 2 in brackets underneath.**

| Opponent | Fast | Normal | Deep |
|---|---|---|---|
| The network itself | 0.63 (0.50) | 0.80 (0.68) | 0.87 (0.80) |
| Looser (T = 2) | 0.57 (0.46) | 0.69 (0.63) | 0.77 (0.75) |
| Sharper (T = 0.5) | 0.64 (0.49) | 0.75 (0.65) | 0.78 (0.74) |
| The exploiter network | 0.56 (0.46) | 0.80 (0.65) | 0.82 (0.74) |
| v1 | 0.55 (0.46) | 0.75 (0.63) | 0.79 (0.71) |

The recommendations hold up against opponents that play differently from the network: 85 to 100% of the gain survives in every world, it never gets near zero, and the recommendation is worse than the network's own move in only 5 to 8% of states. A person might play patterns none of them contain.

## Results

**Simulation only. Opponents are networks and the exact trump-free solver, never people. The search row is a per-decision gain in win-probability points, not a match win rate.**

| Question and conditions | Result |
|---|---|
| v3 network vs v1 network Best move for both, 15,000 matches over five rule sets | 53.07% [52.27, 53.86] |
| Gap to perfect play, no trumps Points of match win rate, mean of 6 cells, 60,000 matches each | v3 0.11, v1 1.59 |
| v3 + search vs v1 + search Search player vs raw v2 and exploiter networks, 2,500 matches each | 58.2 vs 56.8 |
| What search adds per decision 2,000 states, margin rule at 2, judged by an independent search, in the model's own world. Gain in points, for the fast / normal / deep budgets | +0.67 / +0.88 / +0.96 |
| A network trained only to beat it Copy fine-tuned for 3,285 updates against the frozen target, 60,000 matches each | wins 52.48% vs v1, 47.75% vs v3 |

The network on its own got better. With the search switched on, v3 and v1 come out level within noise: the difference is +1.4 points with a standard error of 1.35.

**What does level within noise mean?**

A standard error says how far a measured number could drift if I ran the whole test again with fresh random games. A gap of 1.4 points with a standard error of 1.35 is about one standard error, so the true gap could easily be zero or negative. As a rough rule, anything within two standard errors of zero can't be told apart from luck.

v3 is ahead of v1 on every rule set I tested, by 1.9 to 4.6 points. The exploiter is a copy of the network fine-tuned only to beat one frozen opponent. Against v1 it found 2.5 points. Against v3 the same recipe found no exploit, which says this recipe did not find one, not that none exists. On v1, the fast search beat the plain network in 61.8% of 322 duplicate matches [56.4, 66.9], with the margin at 2, which is why v3 was trained on search labels.

### Step through the seven cycles

Figure: Chart. Each v3 cycle's checkpoint against the frozen v2 baseline, 5,000 matches per point, bars about plus or minus 1.4 points (two standard errors of 0.7). Cycle 6 is the one the advisor runs on. This per-cycle check is a trip wire for regressions, not the verdict. The 15,000-match test in the table is.

Read the 53.07% with care. The opponent is v1, which v3 descends from, and the search labels were made against v1-like play, so part of the gain may be exploiting v1's habits. A player that can see the hidden cards beat v1 83.7% of 182 matches (±2.8). That player isn't perfect, so at most about 16% of matches at this level are unwinnable even with that knowledge. In the no-trump game the numbers are cleaner: a perfect player beats the best fixed rule, drawing while under 16, by winning only 53 to 55 matches in a hundred, and a player who sees the opponent's hidden card reaches 68.

## What I ran into

**PPO levelled off.** v1 beats its own older snapshots by less and less, and it is level with the snapshot from update 20,000. One win or loss after about 80 decisions is a noisy teacher, so v3 moved to search labels.

Figure: Chart. How v1's final network does against its own earlier snapshots, default rules, 3,000 matches per point, with 95% intervals. The gap closes toward 50, meaning the recent snapshots are as good as the last one.

**The search adds less on top of v3.** In match play against v1-like opponents the search is worth about +4 points on top of v3 and +7.5 to +9.8 on top of v1, even though per decision, in the search's own model, the gain is the same (+0.67 against +0.69). A network is a better model of opponents that play like itself, and every opponent here descends from v1. After seven cycles the network also hasn't moved toward the search at the states where the two disagree. Two further variants and weight-averaged checkpoints came out level with v3 (48.5 to 50.4%), and the v4 run was stopped.

**Hand-written rules lowered the win rate.** Forcing "Hush before Stay" onto v1 dropped it to 38.3% in 6,000 matches. Overriding a recurrent network with a rule can disrupt its own plans. Blanket rules for the card game give the same kind of result: in the no-trump game, never drawing from 16, 17 or 18 wins 47.1, 49.2 and 49.5% against plain v1.

**v1 draws too often.** Without trumps the solver knows the right answer, so v1's errors are easy to count: about 19 decisions in 1,000 are clear errors, where the other action was at least 3 points better, and 86% of those are draws that should have been stays. The error grows with the hand total.

Figure: Chart. v1's draws in the game without trumps, 47,532 decisions checked against the exact solver. An error is a draw where staying was at least 3 points of match win probability better.

With trumps the picture is less sharp. A search flags about one decision in ten, and where the exact answer is known, 44% of the flags were real. The flags with a claimed gap of 8 to 15 points held up when I replayed both options from the exact state against the real opponent (+11.4 points on average, 14 cases), and the three above 15 did even better. The 3 to 5 point flags were within noise. The errors concentrate in card-moving trumps (Exchange, Remove, Return, Destroy, Reincarnation) played on the wrong card or at the wrong time. That study covers 4,648 decisions, about a quarter of the size I planned, since I ended the run early. v1 beat three hand-written stand-ins for human players 88 to 95% of the time (79 to 83% with 2 trumps per round). Those are guesses at how people play, not people, and without trumps they tie it.

## Where it stands

How strong it is against a good person is the next thing to find out. I've played one game against it, on 13 October, against v1 with search (not v3), with no hints, and it won in four rounds; one game of something this lucky settles little. Every other number here is against networks, hand-written bots or the exact solver, and the advisor has not yet been used in a real game against a real opponent. The contested rules (Hush, the scope of Shield, Bless) are modelled but not yet confirmed in the real game, and a different reading there would change the advice in those positions. The advisor's win percentage can also differ from the search's by several points in trump-heavy positions (39% against 48% in one test); which of the two is better calibrated is open. Nothing is running right now.

- **Stack**: Rust engine with a PyO3 binding, Python reference engine, PyTorch for training, numba for the solver; trained on an RTX 5070 Ti
- **Status**: Working
- **Code**: Not public

[← All projects](https://oliviercouthaud.com/projects)