Source: https://oliviercouthaud.com/rookery · Status: Prototype · 2026 · Tags: Next.js, FastAPI, GNN

# Rookery

A feed for research papers that works like a social timeline, and the recommender that decides which papers show up in it. I built the backend and all the AI; Roman Slack built the front end.

## What I wanted

Rookery was a social-timeline feed for papers, discovery first and search second, and I made it with Roman Slack. Roman did the front-facing side: the interface and the publicity. I built the whole backend, and the AI part was entirely mine. That meant the API, the database, the data pipeline and the recommender: given a reader, which papers come next. That has to work for a reader with no history at all, for one with three likes, and for one with a hundred, and it has to stay interesting, not just return the nearest neighbours of whatever was liked last.

The corpus was about 110,000 papers pulled from OpenAlex, mostly AI and machine learning, tagged with 3,899 topics. Every paper had a 768-number SPECTER embedding from Semantic Scholar.

**What is a SPECTER embedding?**

SPECTER is a language model trained on scientific papers, using which papers cite which. It turns a paper's title and abstract into a list of 768 numbers, and papers about similar things end up with similar lists. Similarity is then just a dot product between two lists, so you can rank a whole corpus against one paper, or against the average of several.

**What is cold start?**

A recommender learns from what people click, which means it knows nothing about someone brand new. That is the cold-start problem. Rookery was designed to ask new readers to pick topics at sign-up, so the first feed is built from those, and likes take over gradually.

## How it works

Offline, the pipeline pulls papers and citation links from OpenAlex, attaches the SPECTER vectors, and builds a graph of papers, authors and topics linked by citations, authorship, shared references and topic tags. A graph neural network then turns each paper's 768 numbers into 64 new ones that also reflect which papers it sits next to. The vectors go into a FAISS index, so finding the nearest papers to a reader's vector is one fast exact search.

1. OpenAlex. Titles, abstracts, authors, topic tags, citation counts and the links between papers. About 110,000 papers.

Figure: Diagram. The six stages, left to right. Click or tap a stage for a short explanation.

**What is GraphSAGE?**

GraphSAGE is a kind of graph neural network. Each layer rewrites a node's vector by combining it with the average of its neighbours' vectors, so after two layers a paper's new vector carries information from the papers around it and the papers around those. Here that means a paper's vector reflects the citation neighbourhood it sits in, not only its own abstract.

I trained it with a ranking loss on a proxy task: given the papers an author wrote, can you rank the papers that author cited above random ones? An author's profile is the mean of the vectors of their papers. For each cited paper on the training side, the network is pushed to score it higher than up to 5 random papers, by a margin. Citation edges run in both directions.

The reason for that proxy is that I had no readers. Authors cite papers they found worth citing, so what an author wrote and what they cited is a stand-in for a reader and their likes.

Serving builds one vector per reader. It starts from a centroid of the papers in the topics they picked, weighted by PageRank over the citation graph so that well-cited papers count more. Once they have liked something, it adds a second centroid of the liked papers and blends the two. The weight on the likes is, roughly, the number of liked papers divided by 20, capped at 0.8. The candidates are the papers nearest to that blended vector, found with the index.

### How the feed shifts as a reader likes papers

The feed is not only nearest neighbours. Pages are laid out in the repeating order above: eight relevance picks, two recent papers and one surprise. Recent papers are mostly from the last 90 days, matched to the reader's topics or to neighbouring ones. A surprise is a paper from a topic next to one of the reader's, a hidden gem, or something by an author they follow. A hidden gem is a paper with 100 citations or fewer that sits close to the reader's vector.

On top of that, every interaction carries a weight: a like is 1.0, a share 0.9, a bookmark 0.3, a hide -1.0, and so on, and each weight halves every 30 days. A hidden paper never comes back. A bookmark you haven't acted on comes back after 3 days, then 6, 12, 24 days and so on, doubling each time, until you like it, and up to two are slipped into the first page.

## Results

Two tests, and both are proxies. The first is the citation test. For each author I kept a random 70% of the papers they cited on the training side and hid the other 30%. Then I scored the whole corpus against the average vector of their own papers and counted how many of the hidden citations came out in the top ten. Authors needed at least 3 papers and 5 cited papers in the corpus to count, which left 32,869 of them. I scored the first 100 of those.

**Citation test: who counted and how the citations were split (per-author averages)**

| Measure | Value |
|---|---|
| Authors that qualified | 32,869 |
| Papers per author | 5.3 |
| Cited papers, training side | 24.4 |
| Cited papers, hidden side | 11.1 |

This test leaks, so I am not quoting a hit rate from it. The links I hid were still in the citation graph the network passed messages over, so each hidden paper was a direct neighbour of the author's papers, and any score from it would be optimistic. A version with those edges removed is the obvious next test. The 100 authors I scored are also among the authors whose other citations the network trained on, so the test measures how well it fills in the rest of a known author's reading list, and not how it does for an author it has never seen.

The second test is the pretend readers: 100 of them, 10 sessions each, 20 papers shown per session. A pretend reader clicks with a probability set by simple rules, 0.60 if the paper is on one of their topics, 0.75 if it is by an author they follow, 0.05 if it is unrelated, plus small boosts for citations and recency and some random exploring. I ran three versions of the feed through it. Precision here means the share of shown papers that sit in one of the pretend reader's topics.

Figure: Chart. Share of shown papers that match a pretend reader's topics, by session, averaged over 100 pretend readers. The 100% in session 1 for the first version is not something I would read anything into.

**The three feed versions after 10 sessions, 100 pretend readers**

| Measure | First version | Adaptive version | Explore-heavy |
|---|---|---|---|
| Precision | 32.8% | 26.5% | 5.3% |
| Click rate | 30.0% | 26.7% | 12.3% |
| Papers from the last 90 days | 1.0% | 25.0% | 39.1% |
| Surprise clicks | 9.8% | 12.7% | 26.0% |

The first version switched between three methods. New readers got a score from popularity, closeness to their topics and how many of their topics a paper covered. Under 5 likes it used plain SPECTER similarity, and from 5 it used the GraphSAGE vectors. It was the most precise, and it showed almost nothing recent: about 1 paper in 100 from the last 90 days. The adaptive version gives up about 6 points of precision and shows 25 in 100 from the last 90 days, and that was the trade I made. I ran this comparison before the last round of tuning and did not rerun it. The adaptive design is the one I kept.

## What I ran into

The explore-heavy version started every reader with pure exploration and ignored the topics they had declared, and the pretend readers found about 5 in 100 of its papers relevant. The adaptive version, which always starts from the declared topics, came out of that.

In the adaptive version, surprises made up 12.7% of clicks, against a target of at least 15% in that run.

Both tests are loose stand-ins. Authors cite for many reasons, some of them not "I would like more like this", and on top of the leak described above that makes the test a loose stand-in for what a reader wants. The pretend-reader numbers have their own limit: the click rules are my guesses, and the 0.60 for a topic match is a number I picked, not one I measured. That makes those results a check that the feed does what I designed, and not a measure of how real readers would react.

## Where it stands

A waitlist for Rookery was online and the site did not open to the public; it is offline now. No real reader used the feed, so all the testing above was offline, on citations and pretend readers. The next measurement would be against what real readers actually liked, which none of the numbers above include.

The site never opened to the public because the waitlist did not draw enough interest, and I did not have the time to push it next to my other projects. It is offline now. I would have liked to see it live. As a learning project it was a good one: I handled the graph and the graph neural network, and took them as far as I could with analytics aimed at scientific papers, while Roman Slack built the front end and handled publicity.

- **Role**: With Roman Slack. I built the whole backend and all of the AI; Roman built the front end (the interface) and handled publicity.
- **Stack**: Next.js, React, TypeScript, Tailwind; FastAPI, PostgreSQL, Redis; PyTorch Geometric, FAISS
- **Data**: OpenAlex papers and citations, SPECTER vectors from Semantic Scholar
- **Status**: Prototype
- **Online**: A waitlist was online; the site did not open to the public. Offline now.
- **Code**: Not public