Source: https://oliviercouthaud.com/textdetector · Status: Prototype · 2026 · Tags: Python, DistilBERT, TF-IDF

# TextDetector

Guesses a writer's gender and age bracket from text, and measures how the guess improves with the amount of text.

## The question

This started as a random idea: I wanted to see how far a text-only guess of gender could go, and nothing I do uses it. Three things interested me. Is gender detectable from text at all? How good can a guess get? And does more data, or a bigger model, change that? So it is an experiment, not a product. The labels are binary M/F, exactly as the source corpora give them, so what the model measures is the statistical habits of specific online communities, not gender.

## How it works

Training data is five published corpora: the Blog Authorship Corpus, a Twitter set with one tweet per author, a mirror of PAN 2018, and RtGender's Reddit and Fitocracy sets, the last two research-only. Each text is cut into chunks of random length, from tweet-sized to page-sized, so one model sees both. Click or tap any box below to see what it does.

**Five corpora.** Five public collections of writing with a male or female label: the Blog Authorship Corpus (2004 blogs, which also has ages), a set of 200K tweets with one tweet per author, a mirror of the PAN 2018 tweet set (1,900 authors, 100 tweets each), and two RtGender sets, one from Reddit and one from a fitness site called Fitocracy. Together that is about 960K training chunks, or 760K for the linear model, which gets a capped share of the blog. The two RtGender sets are research-only, which matters at the end.

**Split by author.** Each author goes entirely into train, validation or test, 80/10/10, so the model is never tested on someone whose other writing it has seen. Twitter is the exception: that dataset comes with its own split, and every author in it has exactly one tweet anyway.

**Chunker.** Every text is cut into pieces. The target length of each piece is drawn at random from 8, 16, 32, 64, 128 or 256 words, with short ones drawn more often (22% each for 8 and 16 words, down to 8% for 256), and pieces under 5 words are thrown away. This is what lets one model handle a tweet and an essay.

**TF-IDF + SGD logistic.** The linear backend. Word 1-grams and 2-grams plus character 3 to 5-grams, fed into a logistic regression trained with averaged SGD. It trains in minutes on a CPU, and I can read which words it leans on.

**DistilBERT.** The transformer backend, fine-tuned for two epochs on all 960K chunks on a rented RTX 4090, about 30 minutes. It adds about 2 to 3 points over the linear model on blog, PAN18 and Reddit, 1.6 on Fitocracy, and nothing on the one-tweet-per-author Twitter set. It truncates at 256 tokens.

**Raw score.** Both backends hand back the same thing: logit(male) minus logit(female). It is a number, not yet a probability, and it means different things at different lengths, which is why the next box exists.

**Length-binned calibrator.** The part I care about most. A raw score from 10 words and one from 500 words are not worth the same, so each length bin gets its own Platt scaler, fit on the validation split. Both backends use the same calibrator code. Bins: up to 12, 24, 48, 96, 192, 384 words, and above.

**P(male) and the band.** The calibrated number is the probability of male. The tool turns it into words: under 55% it says essentially unsure, up to 65% a slight lean, up to 80% a moderate lean, and above that a strong lean. For a short message the calibrated probability stays close to 50%, so the tool reports low confidence, on purpose.

**Age head.** A separate linear model on the same text. It trains only on the blog labels (three brackets: 13-17, 23-27, 33-48), so it only means anything for blog-like text, and it is not calibrated.

Figure: Diagram. Click or tap a box; the explanation appears under the diagram. Both backends share the same scorer interface and the same calibrator code. * Blog, PAN18, Reddit and Fitocracy are split by author 80/10/10; Twitter uses the dataset's own split. DistilBERT was fine-tuned on a rented RTX 4090; calibration and evaluation ran locally.

Both backends sit behind one interface, so everything after the raw score is shared. From the command line it looks like this. The message is one I made up and the output is an illustration of the format, not a real run.

$ python -m textdetector.predict "ok but we should go to that new cafe this weekend"
backend: transformer
input: 1 message(s), 11 words total
gender: female 56% (slight lean)
age: 13-17 55%, 23-27 32%, 33-48 13%

**What is TF-IDF with a logistic regression?**

TF-IDF counts how often each word or word pair appears in a text and turns down the ones that appear everywhere. Logistic regression then learns one weight per feature, so a positive weight pushes the guess towards male and a negative one towards female. Mine also uses fragments of 3 to 5 characters, which catches spelling and slang. Because the weights are plain numbers, I can list the words it leans on.

**What is DistilBERT?**

A smaller, faster version of BERT, a language model that has already read a large amount of English. I fine-tuned it, which means I kept what it knew about language and trained it for two passes (epochs) over my chunks to output male or female instead. Unlike the linear model, it sees word order and context.

**What is Platt scaling?**

A model's raw score is not a probability: a score of 2 does not mean 88%. Platt scaling fits a small logistic curve on held-out data that maps scores to probabilities, so that when the tool says 70% the aim is that about 70% of such guesses are right. I fit a separate curve per length range, because the same raw score means something different on 10 words than on 500.

**Why split by author?**

If posts from the same person land in both training and test, the model can learn that person's quirks and look much better than it really is. Splitting by author means every test author is a stranger to the model, so the accuracy is what you would get on someone new.

## Results

**Accuracy on held-out authors, chunk level, calibrated at the 0.5 threshold. Linear means TF-IDF + logistic. Chance is 50%.**

| Domain | Linear | DistilBERT |
|---|---|---|
| Blog | 62.9% | 65.9% |
| Twitter, 1 tweet each | 59.4% | 59.3% |
| PAN18, per tweet | 59.9% | 62.0% |
| Reddit | 65.9% | 68.4% |
| Fitocracy | 65.0% | 66.6% |

Length matters more than the model. On blog text DistilBERT goes from 59.7% at 12 words or fewer to 76.1% at the longest bin, and the linear model goes from 58.1% to 72.6%.

Figure: Blog test chunks, accuracy on held-out authors by chunk length. The longest bin is labelled 193-384 words, but chunks stop at 256 words, so in practice it is 193 to 256. The y-axis starts at 50%, which is chance.

### Accuracy by amount of text, in each domain

59.7% / 58.1%

DistilBERT / TF-IDF + logistic, on 10,912 held-out chunks of 12 words or fewer.

**Blog, accuracy by length. Linear means TF-IDF + logistic.**

| Words | DistilBERT | Linear |
|---|---|---|
| 12 or fewer | 59.7% | 58.1% |
| 13 to 24 | 63.7% | 60.4% |
| 25 to 48 | 66.7% | 63.1% |
| 49 to 96 | 71.1% | 67.2% |
| 97 to 192 | 74.0% | 70.8% |
| 193 to 256 | 76.1% | 72.6% |

Black bar, DistilBERT; outlined bar, TF-IDF + logistic; red marks the length you picked. Bars start at 0, the dashed line is chance. Bins with fewer than 200 test chunks are not reported, so short-text domains have fewer bins.

Blog climbs steadily with length. Twitter, which is one tweet per author, does not: DistilBERT stays between 57% and 61% in every bin. Reddit and Fitocracy are mostly short texts, so their longer bins hold few chunks (437 for Fitocracy at 49 to 96 words, 480 for Reddit at 97 to 192), and those bins are noisier.

Figure: PAN18 authors, accuracy by how many tweets are averaged. The k=100 point uses only 83 authors.

Averaging scores over many tweets from one PAN18 author reaches 79.5% at 100 tweets, against 74.7% for the linear model. Published PAN18 winners reach about 82% with heavier ensembles, so this is not a new result.

## Age

Age, in three blog brackets (13-17, 23-27, 33-48), is 56.5% accurate at chunk level with the linear model only. Uniform chance is 33%, but always guessing the largest bracket gives 44.5%. Macro-F1 is 0.51. The oldest bracket is right about 29% of the time, and most errors for the youngest and oldest brackets land in the neighbouring one.

Figure: Each row sums to 100%. The orange diagonal is the correct guess. In the 33-48 row, more than half of those chunks are guessed as 23-27.

## What I ran into

Single tweets from the one-tweet-per-author Twitter set stay at 59 to 60% for both models, and PAN18 tweets score 60 to 62% per chunk. I think that is a ceiling in the data: many short texts carry no gender signal at all.

The 79.5% figure comes from 83 authors, because only those have 100 tweets, which is a standard error of roughly 4 points. The curve is not monotonic (63.2% at 3 tweets, below 64.2% at 1, and 69.5% at 10, below 71.1% at 5). Between model versions the k=100 result moved by a point or two, which is one or two authors.

Adding Reddit and fitness data lowered the transformer's blog accuracy by 0.7 points (66.6% to 65.9%) in exchange for coverage of the other domains. I tried a Bluesky corpus with undocumented auto-labels; it agreed with my model at AUC 0.56, so I left it out. PANDORA needs an access request, and the one open mirror lacks the demographic columns. TwiSty has no English subset.

One GPU run was lost to a data bug. Newer pandas silently drops the grouping column from groupby().apply(), which removed the labels on the remote rebuild. The model had a healthy training loss and chance-level evaluation. I now check content hashes between machines. The rented GPU time came to about 3.5 hours at $0.30/h, including that run.

The linear model's strongest features are "my husband", "hubby" and "knitting" for female, and "my wife", "halo", "chess" and even "the" for male. Mostly it has learned stereotypes, so people who write against type are consistently guessed wrong.

## Where it stands

It was a curiosity and I have no plans to turn it into anything; if I come back to it, it will be to test the several-messages case. The command-line tool concatenates several messages into one text for a joint guess, whereas my author-level numbers average per-tweet scores, so the two are not the same measurement. The concatenation path is not evaluated yet. The transformer also truncates at 256 tokens, which cuts the longest chunks a little. RtGender is research-only, so the trained models are not for release or commercial use.

- **Stack**: Python, scikit-learn, DistilBERT (Hugging Face Transformers), PyTorch on a rented RTX 4090
- **Data**: Blog Authorship Corpus, a one-tweet-per-author set, PAN 2018, RtGender (Reddit, Fitocracy)
- **Status**: Prototype
- **Use**: Research only, and not used for anything. Numbers are from 22 July 2026.
- **Credit**: Datasets credited to their original authors: Blog Authorship Corpus, PAN 2018, RtGender (research use only) and a one-tweet-per-author Twitter set (Apache-2.0)
- **Code**: Not public