Rookery

What it was

Rookery was meant to be a website where research papers turn up in a feed, the way posts do on a social app. The idea was that good papers get lost. Nobody can read everything, and searching only helps if you already know the words to search for. So instead of searching, you would open the site and the papers would come to you.

I made it with Roman Slack, and we split it down the middle. Roman did everything you would see: the site itself, which copies the look of a social feed, and the work of getting the word out. I did everything behind it, the whole backend, and all of the AI that decides which papers you get. That second part is the part I cared about, because a feed is only worth opening if the next thing in it is good. A waitlist was online, but the site itself never opened to anyone, and now the waitlist is offline too. I am not too proud of that.

How it chose papers

Think of a bookshop assistant who has just met you. At first all they know is what you told them at the door, a few subjects you like, so they pick from those shelves. Every time you take a book down they learn a little more, and after a while they listen more to what you took than to what you first said. Rookery was built to work the same way. You would pick topics first, and each paper you liked would move the feed a bit further towards your real taste.

To tell which papers are close to each other, it used the way papers refer to each other. Papers that get cited together tend to be about related things, so a small neural network, a program that learns from examples, learned to place such papers near each other on a kind of map. Liking one paper pulls your feed towards the papers around it on that map.

A feed made only of more of the same gets dull, so some slots were kept back for other things: recent papers, and a few from neighbouring topics or from less-cited papers that still fit your taste.

What happened when I tested it

There was no easy way for me to tell whether a feed is good, so I tested it two ways. First I took authors, hid about a third of the papers they had cited and asked the system to find them again. That test was generous, because the system had already seen those links on its map. Second I built a hundred pretend readers who click on papers according to rules I wrote, and ran them through ten sessions on a few versions of the feed.

That second test is only as good as my rules. It tells me the feed does what I designed it to do, not that a real person would enjoy it. I built a few versions of the feed and compared them. One of them leaned so hard on surprises that by the end only about 5 in 100 of its papers were on a pretend reader's topics. It threw away the topics you had just picked, so that was a mistake. The version I kept took them seriously from the first page.

What's next

Nothing is running. What is missing is a test against what real readers actually liked, because no real reader ever used the feed. Everything I tested came from citations and pretend readers.

What I wanted

Rookery was a social-timeline feed for papers, discovery first and search second, and I made it with Roman Slack. Roman did the front-facing side: the interface and the publicity. I built the whole backend, and the AI part was entirely mine. That meant the API, the database, the data pipeline and the recommender: given a reader, which papers come next. That has to work for a reader with no history at all, for one with three likes, and for one with a hundred, and it has to stay interesting, not just return the nearest neighbours of whatever was liked last.

The corpus was about 110,000 papers pulled from OpenAlex, mostly AI and machine learning, tagged with 3,899 topics. Every paper had a 768-number SPECTER embedding from Semantic Scholar.

What is a SPECTER embedding?

SPECTER is a language model trained on scientific papers, using which papers cite which. It turns a paper's title and abstract into a list of 768 numbers, and papers about similar things end up with similar lists. Similarity is then just a dot product between two lists, so you can rank a whole corpus against one paper, or against the average of several.

What is cold start?

A recommender learns from what people click, which means it knows nothing about someone brand new. That is the cold-start problem. Rookery was designed to ask new readers to pick topics at sign-up, so the first feed is built from those, and likes take over gradually.

How it works

Offline, the pipeline pulls papers and citation links from OpenAlex, attaches the SPECTER vectors, and builds a graph of papers, authors and topics linked by citations, authorship, shared references and topic tags. A graph neural network then turns each paper's 768 numbers into 64 new ones that also reflect which papers it sits next to. The vectors go into a FAISS index, so finding the nearest papers to a reader's vector is one fast exact search.

1. OpenAlex. Titles, abstracts, authors, topic tags, citation counts and the links between papers. About 110,000 papers.

What is GraphSAGE?

GraphSAGE is a kind of graph neural network. Each layer rewrites a node's vector by combining it with the average of its neighbours' vectors, so after two layers a paper's new vector carries information from the papers around it and the papers around those. Here that means a paper's vector reflects the citation neighbourhood it sits in, not only its own abstract.

I trained it with a ranking loss on a proxy task: given the papers an author wrote, can you rank the papers that author cited above random ones? An author's profile is the mean of the vectors of their papers. For each cited paper on the training side, the network is pushed to score it higher than up to 5 random papers, by a margin. Citation edges run in both directions.

The reason for that proxy is that I had no readers. Authors cite papers they found worth citing, so what an author wrote and what they cited is a stand-in for a reader and their likes.

Serving builds one vector per reader. It starts from a centroid of the papers in the topics they picked, weighted by PageRank over the citation graph so that well-cited papers count more. Once they have liked something, it adds a second centroid of the liked papers and blends the two. The weight on the likes is, roughly, the number of liked papers divided by 20, capped at 0.8. The candidates are the papers nearest to that blended vector, found with the index.

The feed is not only nearest neighbours. Pages are laid out in the repeating order above: eight relevance picks, two recent papers and one surprise. Recent papers are mostly from the last 90 days, matched to the reader's topics or to neighbouring ones. A surprise is a paper from a topic next to one of the reader's, a hidden gem, or something by an author they follow. A hidden gem is a paper with 100 citations or fewer that sits close to the reader's vector.

On top of that, every interaction carries a weight: a like is 1.0, a share 0.9, a bookmark 0.3, a hide -1.0, and so on, and each weight halves every 30 days. A hidden paper never comes back. A bookmark you haven't acted on comes back after 3 days, then 6, 12, 24 days and so on, doubling each time, until you like it, and up to two are slipped into the first page.

Results

Two tests, and both are proxies. The first is the citation test. For each author I kept a random 70% of the papers they cited on the training side and hid the other 30%. Then I scored the whole corpus against the average vector of their own papers and counted how many of the hidden citations came out in the top ten. Authors needed at least 3 papers and 5 cited papers in the corpus to count, which left 32,869 of them. I scored the first 100 of those.

Citation test: who counted and how the citations were split (per-author averages)

Measure	Value
Authors that qualified	32,869
Papers per author	5.3
Cited papers, training side	24.4
Cited papers, hidden side	11.1

This test is leaky, so I am not quoting a hit rate from it. The links I hid were still in the citation graph the network passed messages over, so each hidden paper was a direct neighbour of the author's papers. Any score from it would be optimistic, and I never ran a version with those edges removed. The 100 authors I scored are also among the authors whose other citations the network trained on, so the test measures how well it fills in the rest of a known author's reading list. It does not measure how it does for an author it has never seen.

The second test is the pretend readers: 100 of them, 10 sessions each, 20 papers shown per session. A pretend reader clicks with a probability set by simple rules, 0.60 if the paper is on one of their topics, 0.75 if it is by an author they follow, 0.05 if it is unrelated, plus small boosts for citations and recency and some random exploring. I ran three versions of the feed through it. Precision here means the share of shown papers that sit in one of the pretend reader's topics.

The three feed versions after 10 sessions, 100 pretend readers

Measure	First version	Adaptive version	Explore-heavy
Precision	32.8%	26.5%	5.3%
Click rate	30.0%	26.7%	12.3%
Papers from the last 90 days	1.0%	25.0%	39.1%
Surprise clicks	9.8%	12.7%	26.0%

The first version switched between three methods. New readers got a score from popularity, closeness to their topics and how many of their topics a paper covered. Under 5 likes it used plain SPECTER similarity, and from 5 it used the GraphSAGE vectors. It was the most precise, and it showed almost nothing recent: about 1 paper in 100 from the last 90 days. The adaptive version gives up about 6 points of precision and shows 25 in 100 from the last 90 days, and that was the trade I made. I ran this comparison before the last round of tuning and did not rerun it. The adaptive design is the one I kept.

What didn't work

The explore-heavy version was a mistake. It started every reader with pure exploration and ignored the topics they had declared, and the pretend readers found about 5 in 100 of its papers relevant. The adaptive version, which always starts from the declared topics, was the fix.

The adaptive version also missed one of its own targets. In that run I wanted at least 15% of clicks to be surprises and got 12.7%.

The citation test has its limits. Authors cite for many reasons, some of them not "I would like more like this", and on top of the leak described above that makes the test a loose stand-in for what a reader wants. The pretend-reader numbers have their own limit: the click rules are my guesses, and the 0.60 for a topic match is a number I picked, not one I measured. That makes those results a check that the feed does what I designed, and not a measure of how real readers would react.

Where it ended up

A waitlist for Rookery was online, but the site never opened and it is offline now. No real reader ever used the feed, so all the testing above was offline. The missing piece is the one every number above lacks: a test against what real readers actually liked.