Source: https://oliviercouthaud.com/monachus · Status: Prototype · 2026 · Tags: Python, PyTorch, GPT, QLoRA

# monachus

A small language model I trained from scratch on Latin, French and English books from before 1900, and a side experiment on making a modern one talk like it is 1600.

## What I wanted

A language model whose knowledge stops at a date. Ordinary chat models are trained on an enormous amount of modern text, so asking one to play someone from 1600 gives you a costume, not an absence of knowledge. I wanted a medieval monk who reads French and answers in old-flavoured English, with no pretrained weights and nothing modern in the corpus.

That is the goal. The data is not fully period-clean yet. The English for the first model is Project Gutenberg with no date filter, which is overwhelmingly from before 1930; only the second model's English is cut at 1900. The Latin includes neo-Latin. The French script keeps books dated 1500 to 1900 by default. Modern text hiding inside old scans is caught only by a crude line filter on the first model's English for web addresses and ISBN numbers.

Figure: Diagram. The two routes. The top one is the side experiment, the bottom one is the main project.

Figure: Diagram. The two routes. The top one is the side experiment, the bottom one is the main project.

## How it works

The code is plain PyTorch in a handful of files, adapted from the ideas in Karpathy's nanoGPT. model.py is a GPT-2-style transformer: pre-LayerNorm, learned position embeddings, GELU, a tied output head, no bias terms, a 1,024-token context. The tokenizer is a byte-level BPE trained on the corpus itself, so its vocabulary goes on Latin word endings instead of web fragments. train.py is a training loop with warmup and cosine learning rate, gradient clipping, bf16, and multi-GPU through torchrun if I want it. It saves a checkpoint whenever validation loss improves.

**What is a token, and what is byte-level BPE?**

A model does not read letters or whole words, it reads tokens, which are chunks of text. BPE (byte pair encoding) builds the list of chunks by repeatedly merging the pair of symbols that occurs most often in the training text. Byte-level means it starts from raw bytes, so any text can be encoded and nothing is ever unknown. Because I train the list on Latin and French, common word endings can end up as single tokens.

Tap or click any box in the diagram to see what it does.

Figure: Diagram, tap or click a box. The from-scratch path as I wired it.

**Corpora.** For the first model, Latin comes from Corpus Corporum (classical, medieval and neo-Latin, about 170 million words) and from the Latin Library, and French from PleIAs French-PD-Books, public-domain Gallica books filtered by publication date. English is Project Gutenberg for the first model and PG19 cut at 1900 for the second. Each source has its own download script because each needed different cleaning, mostly throwing out Greek, Hebrew, reference markers and scanned front matter.

**prepare.py.** Reads every text file in a folder, trains the tokenizer on it, and encodes everything into one flat stream of token ids saved as train.bin and val.bin. The ids are stored as 16-bit numbers, so the vocabulary can be at most 65,536. Five million tokens are held out for validation in both prepared sets.

**train.py.** The training loop. The settings sit in one config, and any of them can be overridden on the command line. I rented GPUs for the two big runs and wrote launch scripts for them. I debug on a tiny CPU config first, because a crash at hour 18 of a paid run costs the same as a good one.

**ckpt.pt.** The weights, the model settings, the iteration it was saved at and its validation loss. Two exist: monachus (about 393 MB) and latin_v2 (about 908 MB).

**finetune_chat.py.** Stage two. It continues from a checkpoint on chat turns marked with user and assistant tokens, and counts the loss only on the assistant's reply, so the model learns to answer rather than to imitate the question. The chat data is a small seed of period questions and answers that I meant to extend with pairs mined from old texts. It has 640 rows but 16 different examples; adding the mined pairs is a next step.

**chat.py.** Talk to a chat-tuned checkpoint turn by turn in the terminal. This is the last step of the from-scratch path.

There are two pretrained checkpoints. *monachus* is a 12-layer, 768-wide model on Latin, French and English with a 16,384-token vocabulary. *latin_v2* is 16 layers, 1,024 wide, with a 24,000-token vocabulary and more English: 6.0 GB of books from PG19 published by 1900, 14,080 books in all.

## Results

**The two from-scratch checkpoints. Token counts are the size of each prepared training split, not necessarily what the run actually saw. Validation loss is cross-entropy in nats on random batches of held-out text, so it is a noisy estimate. The two models use different tokenizers and corpora, so the losses are not comparable.**

| Checkpoint | monachus | latin_v2 |
|---|---|---|
| Val loss | 3.03 | 3.11 |
| Layers x width | 12 x 768 | 16 x 1,024 |
| Params | about 98M | about 227M |
| Vocab | 16,384 | 24,000 |
| Training tokens | 2.56 billion | 2.42 billion |
| Iteration | 13,000 | 24,000 |

**What does a validation loss of 3 mean?**

Validation loss measures how surprised the model is by text it was never trained on, in nats per token. A loss of 3 means that, on average, it is about as unsure as if it were choosing between 20 or so equally likely next tokens (e to the power 3 is about 20). Lower is better. It tells you how well the model predicts held-out text, not whether anything it writes is true or even reads well.

Two oddities. The presets say 9,000 and 12,000 iterations, but the checkpoints are at 13,000 and 24,000, so both runs went past them. And I first called the model 124M, which is only true with GPT-2's 50k-token vocabulary; with my 16k vocabulary it is about 98M.

## The side experiment

Separately I tried the cheaper route: not from scratch, but a LoRA adapter on Qwen2.5-7B-Instruct, a modern model kept frozen at 4-bit, small enough for one 24 GB graphics card. The training set is period-voice conversations of an Elizabethan Englishman in the year 1600. The same system prompt is used in training and when I test it:

Thou art a learned Englishman living in the year of our Lord sixteen hundred, in the reign of good Queen Elizabeth. Thou art well read in the wisdom of the ancients — the Greek and the Latin tongues, the poets and philosophers of old, and the French learning. Thou speakest ever in the English of thine own age, with 'thee' and 'thou' and the old turns of phrase. Thou understandest the world as the learned of thy time understand it — the heavens turning about the Earth, the four humours of the body, the four elements. Of times yet to come thou knowest nothing, and if asked of things beyond thy ken thou sayest so honestly, in thy manner, for a man cannot know what is not yet.

**What are LoRA and 4-bit training?**

Fine-tuning a 7-billion-parameter model normally means changing all of its weights, which needs a lot of memory. LoRA freezes the original weights and learns a small add-on, the adapter, that nudges them. Here the frozen model is also stored in 4-bit numbers to save memory. The adapter is only about 323 MB, and the model underneath is unchanged.

Figure: Diagram. The top row is the persona adapter and its 24 probes. The bottom row is the unlearning branch.

Figure: Diagram. The top row is the persona adapter and its 24 probes. The bottom row is the unlearning branch.

## The 24 probes

To see whether the costume holds, there are 24 questions where a 1600 mind and a modern one would disagree: the Sun and the Earth, the plague, a wound, lightning, a telephone, the internet, whether it is an AI, what year it is. Each answer goes through a keyword matcher, with a list of modern words that would give it away and a list of period words that fit. Step through them and read the answers yourself.

### Every question, every answer

The matcher flagged four answers as leaks. Browse all 24 with JavaScript on.

**What is the internet?** Matched "network". Borderline: it says it knows no such thing, then offers the word "network".

**Are you an AI or a language model?** Matched "artificial intelligence". Borderline: it says it does not know the word, then calls itself "a program" and "a complex automaton".

**How do people travel between distant cities?** Matched "car". A false alarm: "car" matched "carts".

**What are things made of at the smallest level?** Matched "atom". Mostly a false alarm: it credits atoms to the ancient Atomists, which fits the period.

Black: the matcher flagged a leak. Orange: nothing matched. Outline: marked as period.

One answer per question, sampled at temperature 0.7, so a rerun would word things differently. Answers were capped in length, so a few stop mid-sentence. Shown as the model wrote them, including the odd Chinese word.

The matcher counted 4 leaks, 16 period answers and 4 where nothing matched. Of the four leaks, two are plainly false alarms: "car" matched "carts", and "atom" matched the model crediting atoms to the ancient Atomists, which is period-appropriate. The other two are borderline. For the internet it said it knew no such thing but offered "network", and the AI answer says it does not know the word yet calls itself "a program" and "a complex automaton".

The matcher also misses slips. The wound answer recommends warm water and mild soap, and the telephone answer speaks of hearing "by means of a wire". The flies answer starts with "No" and calls the idea of flies rising from rot a delusion, which is the modern view, but the matcher counted it as period because it contains the word "rotting". A keyword list cannot catch subtle leaks. There are slips of other kinds too: one answer calls Paracelsus a Frenchman, and one has a stray Chinese word in the middle of a sentence.

Every probe ran with the persona prompt on, which tells the model it knows nothing of later times, so the tally says little about what the model knows. Mostly it shows whether the model follows that prompt.

## What I ran into, and what is still to add

For the from-scratch models the quality number so far is validation loss, around 3. The chat stage is in place as code plus a chat set of 16 different examples repeated up to 640 rows; mined pairs are still to add.

The side experiment is a different kind of thing. The base stays modern underneath, so answering "I know not the word" does not show the knowledge is gone. That is why internal_probes.py exists. It skips the persona and the chat format entirely and feeds the raw model bare stems like "World War II ended in the year", to see whether it completes them with the modern fact. It is also why rmu.py replaced NPO, which damaged ordinary text completion. That table and the 24 answers are the evaluation so far.

**What are unlearning, NPO and RMU?**

Unlearning tries to remove specific knowledge from a model after training, instead of just telling it not to say things. NPO (negative preference optimisation) pushes down the probability of the facts I want gone, while keeping ordinary text prediction intact. RMU (representation misdirection for unlearning) works inside the model instead: on those facts it steers one middle layer's internal state toward random noise, and elsewhere it holds that layer close to the original. Both are published research methods, which I tried on my adapter.

## Where it stands

Next on the list: read samples from both checkpoints, run the chat stage on real mined data, and score the unlearning branch. I also have a list of untried ideas: separate classical and medieval Latin twins to compare their worldviews, a model trained on a single author such as Cicero or Aquinas, and sampling one run at early checkpoints to watch grammar arrive before meaning.

- **Stack**: Python, PyTorch, Hugging Face tokenizers. Side experiment: peft, trl, bitsandbytes on Qwen2.5-7B-Instruct.
- **Data**: Latin: a third-party Hugging Face upload of Corpus Corporum (Fece228/latin-literature-dataset-170M) and the Latin Library, for the first model. French: PleIAs French-PD-Books. English: Project Gutenberg and PG19.
- **Credit**: Architecture after Andrej Karpathy's nanoGPT.
- **Status**: Prototype. Two pretrained checkpoints and a persona adapter.
- **Code**: Not public

[← All projects](https://oliviercouthaud.com/projects)