Codenames

I got into this after reading about Gibberlink and the stories of AIs inventing secret languages to talk to each other. I wanted to know whether that would happen in a game, with two small language models, the kind of program that sits behind a chatbot, and nothing to go on except the score. Would they drift into a private code, like two friends whose shorthand nobody else can follow?

The game is Codenames, a word game. Twenty-five words lie face up on a table. One player secretly knows which of them belong to their team, and gives a one-word clue plus a number. "Water, 3" means three of the words have something to do with water. The partner guesses. Pick a word that belongs to the other team and you help them. Pick the assassin and you lose on the spot.

I let the clue be any string of letters, not just real words, so nothing stopped them from inventing a code. I saw no sign of one in the runs I did.

Their clues stayed recognisable English, even when I made secrecy cost them points. Early on they played worse than random guessing, mostly because of mistakes in how I had set the game up. Once those were fixed, they still only got anywhere after I first taught them the game by copying a simple word-similarity player; from that start they found about six of the nine words per game, and starting from nothing they stayed around random.

So I changed the question. A good clue giver does not pick the most accurate clue, it picks the one its particular partner will understand. I built one that works out what a human partner is likely to guess, using surveys where people wrote down the first word a word made them think of, and later about five thousand real human guesses from a public Codenames dataset. For every candidate clue it imagines the partner guessing, and picks the clue most likely to keep the game winnable.

Against a simulated partner built from the survey data, it cleared the board in all 120 games I checked. The catch is that the simulated partner and the clue giver draw on the same data, so it only shows the agent can predict a human-shaped partner, not that it beats a person. The same clue giver with a wrong picture of its partner won about 2 games in 100.

The only test with a real person was small, and the person was me. I guessed on 20 boards without knowing which words the clues were aimed at. I found 32 of the 50, fully solved 7 of the boards, and never touched the assassin. Some of its clues made sense straight away. Others made no sense to me at all, which was odd, and honestly kind of cool to see. I also made a small web game where the machine gives you the clues. It runs entirely in the browser, but I have not put it online because it contains the commercial game's word list.

What's next: playing a lot more real games against many different people, since so far I am the only one who has tried it, and giving the two language models separate brains instead of one shared one. I think the shared brain is why they never split off a secret language, but I have not tested that.

What I wanted

This started with Gibberlink and the stories of AIs inventing secret languages to talk to each other. I wanted to see whether two small language models, rewarded only by the game score, would do the same and drift into a private code. In Codenames one player gives a one-word clue and a number, and a partner guesses which board words it means. I made the clue a stream of lowercase letters with no vocabulary behind it, so nothing forced it to be English, and let the pair play each other thousands of times.

It is the single-team version. Nine words are mine, eight belong to a team that never plays, seven are bystanders and one is the assassin. A team word is worth +1, the other team's card −1, a bystander −0.3 and the assassin −4. Clearing the board is +3 and every turn costs 0.15. I built it over a few days in late July 2026 on rented RTX 4090s, in two halves: three rounds of language-model experiments, then an agent that drops the language model and models the person it is playing with.

Two models and a letter channel

A spymaster and a guesser shared one base model (Qwen3, 0.6B in round one and 1.7B after) and one LoRA adapter, and were trained together with GRPO. To test for a private code, a frozen copy of the untrained model read the same clues as a stranger would. If the pair had invented something, its clues should work with its own partner and fall apart with the stranger.

What are GRPO and LoRA?

GRPO is a reinforcement-learning method. The model plays a batch of games and is pushed toward the plays that scored better than the batch average, without training a separate scoring network. A LoRA adapter is a small set of extra weights attached to a frozen language model, so training changes a small fraction of its numbers. Here the spymaster and the guesser used the very same adapter.

Round one used the 0.6B model for 15 and 18 updates of 128 games each, one run with a KL anchor and one free to wander. Both did worse than random guessing, and most of that was my interface, not the learning.

Offering the guesser a "no guess" option made it pass between 80 and 98 percent of the time, because a generic phrase outscores any specific board word. Letting it write its guess one token at a time committed it to a first token it could not back out of, so I switched to scoring every remaining board word in one batch. Forbidding a clue to end inside a board word made it pad with repeated letters, so now I reject the clue and sample again. And "pass" is itself a word in the deck, so using it to stop quietly revealed that card on 6.5% of boards.

What is a KL anchor?

It is a penalty for moving the model's word choices away from where the original model started. With the anchor on, the clues are held near ordinary English. With it off, nothing but the score stops them drifting somewhere strange, which is what you would need for a private code.

Round two used the 1.7B model with all of that fixed, and ran the comparison I actually cared about: four runs, with or without a warm start and with or without the KL anchor. The warm start was behaviour cloning from a GloVe word-vector oracle on 120 boards. The oracle scores a clue by how much closer its weakest intended word sits than the most dangerous other card, with the assassin weighted hardest. Same RL, same budget: the warm runs reached 5.84 and 6.06 words per game and the cold runs 2.30 and 2.83. Self-play sharpened a pair that already played and could not create one.

Round two was a weak test of the private-code idea, because plain English clues already worked. So in round three I made English expensive. Every guess was also scored by the frozen stranger, and the pair was charged for the probability the stranger put on the word its own guesser picked. If a private code was forming, that number had to fall.

Team words found per game, out of 9, pair against itself on held-out boards (48 in round one, 64 in the others). Compare within a row only.

Round	Untrained	Trained	Random
1 (0.6B)	0.73	0.29, 0.10	2.13
2 (1.7B)	2.95	6.06, 2.83	2.40
3 (1.7B)	2.97	6.41, 6.06	–

Trained, round one: two variants. Round two: 6.06 is after a warm start (no KL arm), 2.83 is the best cold arm. Round three: 6.41 is the control, 6.06 the run paid for secrecy. Random guessing was not measured in round three. Rounds differ in model size (Qwen3 0.6B vs 1.7B), interface fixes, warm start and the secrecy term.

The warm start alone, with no RL, scored 5.48 in round two and 5.64 in round three.

Here are the four round-two runs while they trained. The table below gives the first, best and last update of each run; with scripts on, a slider reads off any single update. These numbers are sampled at temperature 1.0 during training, so they sit lower than the held-out scores in the table.

If a private code were forming, that line would fall. It does not. Averaged over the first ten updates and the last ten, the stranger's probability in the run that was charged for it went from 0.109 to 0.112. In the control run, which was never charged, it drifted up from 0.116 to 0.133. So the charge stopped the clues getting easier to read and never made them harder. Cross-play says the same: against the frozen guesser the secrecy run found 3.56 words, the control 3.36 and the warm start with no RL at all 3.34. Their clues stayed recognisable English, with about 55% of them a single whole-word token in the model's vocabulary.

Searching against a model of the partner

After round three I dropped the language model from the loop. Whatever the pair learned was graded by a guesser that was itself a language model, and every stand-in partner I measured against flattered the system. The next agents work from a model of the partner and search for the clue that partner is most likely to follow.

The first versions scored clues with sentence encoders, which measure similarity of meaning, and stuck at about 6 of 9 words on English. Codenames does not ask whether two words mean the same thing, it asks whether one makes you think of the other. Swapping in ConceptNet Numberbatch, which is built on a graph of how words relate, raised the mean cross-play score over five word packs (English, French, Spanish, invented words and a pack of nonsense strings) from 4.71 to 5.38 words, and Spanish alone by 1.8.

What are Numberbatch and the USF norms?

Numberbatch is a set of word vectors built from ConceptNet, a hand-made graph of how words relate, so it knows that two words go together even when they do not mean the same thing. The USF free-association norms come from a survey of about 5,000 cue words, where people wrote down the first word each one made them think of, which is close to what a Codenames guess is.

As a stand-in for a person I used the USF norms directly: a partner that ranks board words by how strongly people associated them with the clue, with an encoder to fall back on for the 3.4% of words the norms never recorded.

The agent only won about a third of its games. Nine words in eight turns needs 1.125 words per turn and it managed 1.04, so it kept finishing 8 to 9. Every board I tried turned out to be winnable: for every team word on every board, some clue in the vocabulary makes the partner pick that word first. The agent just never looked at it, because it only scored 40 candidate clues picked by a similarity heuristic. Ranking the partner over all 8,000 or so legal clues is a single matrix product, and searching all of them closed the gap.

Against the human-norms partner, 40 to 120 games per row. The last row also lets the clue giver model the partner with the same association norms the partner is made from.

Stage	Words of 9	Win rate
Encoders only, shortlist search	6.22	18%
+ association graph (blend)	6.90	28%
+ shortlist search	7.38	38%
+ exhaustive search over the whole vocabulary	9.00	100%

I checked it on three fresh seeds, 40 games each. The first run won 119 of 120, and the one loss was an assassin hit. Then I raised the assassin penalty and fixed two bugs, and the same seeds went to 120 of 120. That last step is tuned on those seeds, and 120 of 120 is consistent with a true win rate as low as about 97%. It is also the least interesting number here, because of what comes next.

It is the same agent in every row below. Pick one to see what happens when its picture of the partner is right and when it is wrong.

That pair of numbers is the real result. Skill at this game turned out to be almost entirely the accuracy of your model of the other player. Optimising hard against a model multiplies whatever the model gets right, and multiplies what it gets wrong just as hard.

A partner fitted to real guesses

The next step was to stop picking weights by eye. The salt-nlp Codenames Duet data has real human guesses, and after cleaning I had 5,453 turns to fit on and 307 held out. The partner model is a multinomial logit over the board words. Its features are Numberbatch similarity, three sentence encoders, USF free-association norms and a hint prior built from crossword clues.

What is a multinomial logit?

For each word on the board it adds up the features, each multiplied by a fitted weight, and turns the totals into probabilities that sum to one. Fitting means nudging the weights until the probability given to the word a human really picked is as high as possible. A temperature is then tuned so the probabilities are honest numbers and not just a ranking.

Here is the agent that works from that model. Each part is explained under the diagram, and with scripts on you can tap or click a box (or a button below it) to read just that one.

Numberbatch Encoders USF norms Crossword prior LearnedPartner EVSpymaster Clue + count Web export Web game

Numberbatch. Word vectors built on a graph of how words relate. They know that two words go together even when they do not mean the same thing, which is most of what a Codenames clue trades on.

Three sentence encoders. Small multilingual models (MiniLM, MPNet and DistilUSE) that measure similarity of meaning. That is not quite what a guess is, but they can score any string at all, including words I invented.

USF association norms. A survey of about 5,000 cue words where people wrote down the first word each one made them think of, and how often. It is the closest thing I had to how people really associate.

Crossword hint prior. Millions of crossword clues boiled down to which words people use to hint at which answer. It is the only source here about what a person would say, not just what they would think of.

LearnedPartner. For each word on the board it combines twelve features with fitted weights and gives the probability a person picks it. The weights come from 5,453 real human guesses, and a temperature is tuned so the probabilities mean what they say.

EVSpymaster. Takes a cheap first pass over every candidate clue, keeps the best 120, then samples the partner's picks 96 times for each clue and clue size. A clue is scored by the chance the team can still win afterwards, times the chance of missing the assassin, and it assumes the partner stops guessing when it is not confident enough.

Clue and count. The count is the number of words the clue was chosen for. An earlier version asked the model to guess its own count after the fact, so what I meant and what I said could disagree.

Web export. Computes the partner's scores for 3,200 candidate clues against every word in the pack and squeezes them to 8-bit integers, so a web page can carry the whole model.

Web game. One web page where the same search runs in JavaScript and you play the guesser against it. I have not put it online because it carries the commercial game's word list.

One design point mattered more than the rest. The spymaster used to pick the clue whose best-ranked word was a target, and that rewards a clue like soap for a card that carries almost no probability. Choosing by expected score under the calibrated distribution is meant to remove that exploit.

First, how well the fitted partner predicts people. On 307 held-out human turns, with chance at 4.5%, the fitted model beats each of the features it is built from. On pairs it has never seen, though, an LLM asked directly beat it (0.400 against 0.357).

Top-1 accuracy at predicting the word a human actually guessed. Claude Haiku may have seen this public dataset in training. The last column counts only clue and word pairs that never appear in the training data. About 53% of the test set is memorisable, which is why a lookup table scores so well overall and almost nothing on unseen pairs. A dash means not measured.

Scorer	Top-1	Unseen pairs
One encoder (MiniLM)	0.518	–
Encoder consensus	0.573	–
USF human norms	0.599	–
Fitted partner model (12 features, English)	0.664	0.357
Lookup of training pairs	0.616	0.052
Qwen3-8B log-probability	0.349	0.191
Claude Haiku, asked which board word the clue points at	0.661	0.400

One person, 20 boards

The one check that is not a stand-in is a person, and the person was me. I guessed on 20 boards, with clues from this agent as it was on 30 July, without knowing which words it was aiming at. The clues announced 2.5 words on average. I found 32 of the 50 intended words, fully solved 7 boards and never hit the assassin. Of my 44 guesses, 38 were team words, 3 were the other team's cards and 3 were bystanders. Some of the clues made sense at once and some made no sense to me at all, which was odd and a bit cool. Here are ten of the twenty, so you can see the kind of miss.

What didn't work

I saw no sign of a private code in any round. Round one was too short and weak to test it, and round three tried one secrecy weight (3) for about 100 updates, so this does not prove such a code cannot form. The charge cost 0.35 words per game (6.41 for the control, 6.06 for the charged run) and bought no secrecy. My guess is that the shared adapter ties the two roles together: to become unreadable to the base model the spymaster has to move away from it, but the guesser is that same base model plus the same LoRA, so it moves too. I have not tested that.

An earlier, differently rewarded run of round one found a reward hack on its own. The spymaster said "blank" every turn and the guesser passed every time, and the reward improved from −2.36 to −0.94 for doing nothing. Charging for every team word still hidden at the end fixed it, because doing nothing became the worst result. In round two every run peaked part-way and then declined, so the end-of-run numbers in the table understate every run. From round three I picked checkpoints by held-out games and not by training reward.

Measurement kept flattering the system. A human-norms partner with an encoder fallback shared a model with the agent and inflated the result by 1.9 words (8.78 against 6.88 once I rebuilt it). A search-based agent scored 3.33 words higher in self-play than against a held-out encoder, averaged over five word packs. And the exhaustive-search agent with a wrong model of its partner won 2% of games or fewer.

A language model as clue generator added nothing: 4.71 mean cross-play words with it and 4.71 without. Its invented clues were dominated by a frequency-ranked word list, at roughly ten times the compute. Adapting to an individual partner online moved prediction by less than half a percentage point.

The person test is a very small sample, one person and 20 boards, and I do not know how the agent holds up against people who guess differently from me or from the data it was fitted on.

Unfinished

I built two more blind tests aimed at that gap, one where the clue giver picks the target set first and one where a language model vetoes weak clues, but I do not have results for them. Separate adapters for the two language-model roles, and association data for languages other than English, are untried. The next real step is playing a lot more games with people other than me.