Source: https://oliviercouthaud.com/screentranslator · Status: Prototype · 2026 · Tags: Python, OCR, NLLB-200, PyQt5

# ScreenTranslator

A Windows tool that reads text off part of your screen, translates it on your own machine and draws the translation on top.

## What I wanted

I wanted to read German documents on a screen share as they come up, live. It's the visual version of live audio translation, or Google Translate's camera mode without the camera: the translation appears on the screen itself, over the original. A screen share is video, so the text can't be selected or copied, and the same is true of subtitles burned into a film, words inside an image, or apps that draw their own letters. So you drag a box over the text and the translation is drawn in place.

I also didn't want my screen going to a cloud service, so everything runs on my PC. The trade-off is quality: the local model is decent, and Google Translate and DeepL translate better.

It started from Translumo, an open-source screen translator (Apache-2.0). I ported its text-similarity code, wrote my preprocessing to match one of its image helpers, and took from it the idea of matching OCR output loosely instead of exactly.

## How it works

One background thread runs the loop below. The Qt window on the main thread only draws what the thread hands it. Step through one pass with the buttons.

Figure: Diagram. One pass of the loop, in the order it runs. Preprocess, the fuzzy cache, auto-clear and the multi-monitor capture mapping exist only in my local version; the version on GitHub has an exact-match cache and a fixed 500 ms wait.

### Step through one pass

Stage 1. BetterCam, a fast Windows screen grabber, takes the picture, and mss is the fallback if it fails. The area is the whole monitor, a box I drag out, or a window found by its title. With several monitors it works out which one the region's centre sits on and converts to that monitor's own coordinates.

Click a box in the diagram, or use the buttons.

I use Windows OCR because it's built in, needs no model download and gives back lines with word boxes. If Windows doesn't support the language I ask for, it falls back to the languages in my Windows profile. For translation I use Meta's NLLB-200 through Hugging Face Transformers, on the GPU when there is one. It's the only model the tool downloads and runs itself.

**What is NLLB-200?**

No Language Left Behind, a translation model from Meta AI that handles about 200 languages in a single network, so any supported pair works without a separate model for each. I use the distilled 600M version: a smaller copy trained to imitate a bigger one, with 600 million parameters. The download is about 2.4 GB.

The overlay is meant to look like the original with different words, so I estimate a background and a text colour for every block. The background is the median of a ring of pixels around the block, and the text colour comes from sorting the pixels inside it into two groups.

**What is k-means?**

A way of sorting pixels into groups by colour. Pick two starting colours, give every pixel to the nearer one, move each colour to the average of its pixels, and repeat. With two groups on a block of text you end up with roughly "ink" and "paper".

Figure: Schematic, not a screenshot. From OCR lines to the overlay, using the rules I wrote.

The overlay is a single transparent Qt window across all monitors. Windows extended styles make it click-through and keep it from stealing focus, and it's drawn in two passes, all the backgrounds first and all the text second, so one block's text can't be covered by another block's background. The translation is clipped to its backing rectangle.

**Why keep the overlay out of the screenshot?**

The tool photographs the screen, and the overlay sits on the screen. Without a fix it would photograph its own translation, read it as new text and translate that, forever. Windows has a call, SetWindowDisplayAffinity, that tells the system to leave a window out of captures. Some capture tools may still pick the overlay up anyway.

## Why the cache is loose

The OCR doesn't always read the same pixels the same way. The example I test with is a lowercase l in "lazy" coming back as a capital I. With an exact-match cache that's a new string: it gets translated again and can come out slightly different, which would make the overlay flicker. The version on GitHub only matches exactly.

So in my local version the cache compares loosely. Every new block is scored against what's stored for the same language pair, and a score above 0.65 counts as the same text and reuses the stored translation. For Chinese, Japanese, Korean, Thai, Burmese, Khmer, Lao and Vietnamese the bar is lower, 0.55. The cache forgets everything after 3 seconds without a lookup. A second check works on the output: the joined translation is compared with the last six, and a Jaro score above 0.955 counts as a repeat. That check currently has no effect, because the overlay has already been cleared by then (stage 9).

**What are Jaro and Dice?**

Two ways to score how alike two strings are, from 0 to 1. Jaro counts the characters the strings share in roughly the same places, and penalises swapped ones. Dice splits both strings into pairs of neighbouring letters and counts how many pairs they share. I use the average of the two, the way Translumo does, which makes a single wrong letter cost very little.

Figure: Chart. The blended score for the sentence "The quick brown fox jumps over the lazy dog" against four variants, all of which I use as test cases. Shaded: the range where the stored translation is reused.

### Try the matcher

0.968, reuse

Jaro 0.984, Dice 0.952. Read as two translated outputs, only Jaro counts: 0.984 is above 0.955, a repeat by the output check.

This is the same formula the tool uses, written again in JavaScript here. The default threshold is 0.65, and 0.55 for the Asian-script languages.

## How often it looks

On GitHub the loop wakes every 500 ms. In my local version the wait depends on what just happened. After a pass that found nothing, or a screen that hadn't changed, it waits 115 ms. After a full pass of OCR and translation it waits 320 ms. When the new OCR text is similar to the last (between 0.5 and 0.95) and no shorter, it looks like typing or scrolling, and the wait is 620 ms. Time spent working counts against the wait, and there's a floor of 50 ms.

### Set the dial

waits 115 ms

Targets: 115 ms with nothing new on screen, 320 ms after a translation, 620 ms while text is still arriving, with a 50 ms floor.

Illustrative: the three targets and the floor come from the tool's settings, the time the pass took is yours to choose. A pass where nothing changed is only a frame comparison; a pass with OCR and translation is the long one.

## What I ran into

Source language "auto" is not detection yet. In the tool it means English, for both the OCR and the translator, so any other language needs the source language set by hand.

When translation throws an error, the engine logs it and returns the original text, so the overlay shows untranslated words without a visible notice. Input is cut at 128 tokens, and a translation more than three times longer than its source is truncated on screen.

The loose cache has a trade-off. A sentence cut off by scrolling still scores 0.858 against the whole sentence, so it reuses the whole sentence's old translation. The binarisation cut-off is a fixed 150. Paragraph merging compares each line with the next one down, so text in side-by-side columns stays in separate lines.

The moment the screen changes, the old translation is wiped, so there is a blank while the new one is worked out. And the overlay may still show up in some capture tools, even with the capture exclusion on.

## Where it stands

GitHub has my first version, from February 2026. It does not start as pushed, because the language list and settings modules were not part of that upload, so it stops on import. The fuzzy cache, the similarity code, adaptive polling, preprocessing, auto-clear and the multi-monitor BetterCam mapping are on my own machine and are the next thing to push. One set of tests covers the similarity maths and the cache; end-to-end tests of the pipeline are still to be written. It runs on Windows.

- **Stack**: Python on Windows 10/11, Windows.Media.Ocr (WinRT), NLLB-200 distilled 600M via Hugging Face Transformers and PyTorch, PyQt5, OpenCV, BetterCam, mss
- **Status**: Prototype
- **Code**: [github.com/Forkei/ScreenTranslator](https://github.com/Forkei/ScreenTranslator) (public; the newer cache and polling work is not pushed yet)