Source: https://oliviercouthaud.com/simuverse · Status: Prototype · 2025 · Tags: Python, LLM agents, Weaviate

# Simu-Verse

AI characters that each have a second AI, a "subconscious", writing down their memories and handing the useful ones back.

Simu-Verse was meant to be a world where LLM agents move and talk in 3D. I led a team of four. My teammates worked on the environment and the 3D side, and on some of the code, and I took the memory. What I wrote is public as Simu-Verse_Memory: the agent loop, the memory system and a 2D grid that stands in for the world, because the 3D work lives elsewhere. Some of it may sit in a teammate's own repositories. Unity was the plan for the 3D part. It ran from February to April 2025.

## What I wanted

A language model forgets everything between calls. If agents are going to hold relationships or remember where things are, something has to decide what is worth keeping and what to bring back. I didn't want the agent doing that itself, with a database tool it has to remember to call. So each agent gets a paired "subconscious". It writes the memories, and before each turn it picks a few and hands them over as flashbacks. The agent never queries the database, and it never knows the database exists.

## How it works

Every turn makes three kinds of LLM call. The first asks the subconscious for up to three search queries, written as XML. The second is the agent's own reply, also XML: a short reflection and one tool call. The third summarises the exchange into a memory: a summary, one of eight categories, keywords, "critical information" and an importance from 1 to 10. Step through the turn below to see each step and the format it uses.

Step 1, retrieve. The subconscious reads the recent conversation and writes up to three search queries. Each is keyword, semantic or hybrid, with optional filters on category or importance. The main agent is not involved.

Figure: Diagram. One agent turn as the newest version does it; the retry loop is not in the earlier ones. Click or tap a box, or a button, for an explanation. Names follow the code (AgentManager.process_agent_turn, SubconsciousAgent, LLMManager).

### Format at each step

<memory_queries>
  <query>
    <search_type>hybrid</search_type>
    <keywords>meeting, deadline</keywords>
    <query_text>the project deadline</query_text>
  </query>
</memory_queries>

Tap or click a box in the diagram, or use Back and Next. Formats are the ones in the prompts; the values are placeholders.

The LLMManager talks to Ollama by default (the local model is qwen2.5:14b), or to Anthropic or OpenAI. The agent never gets a database tool. That was the point, and it is also why a bad retrieval silently hurts it: it can't ask again.

Each memory is stored in a collection per agent, in Weaviate, a vector database. Only the summary is embedded. The rest, category, keywords, critical information, importance, timestamp, location, agent and id, are stored as plain fields.

**What is a vector database, and what does "embedded" mean?**

An embedding turns a piece of text into a long list of numbers so that texts with similar meaning end up with similar numbers. A vector database like Weaviate stores those lists and can find the stored texts closest in meaning to a new one. That is how "semantic" search can find a memory about "meeting the cafe owner" when the query says "who did I talk to this morning". Here only the summary is turned into numbers.

Figure: Diagram. The memory record and the three ways to search it. In the newest version the record is eight fields plus the object id, and the cut-off is five. Without a Weaviate server the code falls back to a mock store, where "semantic" search is only a substring match.

**What is hybrid search?**

Keyword search (BM25) rewards texts that contain the exact words you typed. Semantic search rewards texts that mean something similar, even with different words. Hybrid runs both and blends the scores. The alpha value sets the blend: 0 is all keyword, 1 is all semantic, and I used 0.5.

## The grid demo

Since the 3D world isn't part of what I published, I made a 2D one with Dash and Cytoscape to run the agents in. Four agents (James, Jade, Jesse and Jamal) stand on a grid running from 50 to 550 in each direction, with three landmarks and four items to walk to or use. Each agent is linked to its nearest neighbour, with no range limit, and the links redraw as they move. Tap or click anything in the picture, or pick it from the list under it, to see what it is.

The grid has four agents (James, Jade, Jesse, Jamal), three landmarks (center_fountain, north_bench, west_cafe) and four items (red_ball, info_kiosk, lab_door, cafe_chair).

Figure: Diagram, drawn from the start positions in the demo. The lines are each agent's link to its nearest neighbour at the start; they move with the agents.

### Pick a marker

The descriptions come from the demo itself: the agent profiles and the landmark and item definitions.

Each agent has a personality-strength slider. It sets how loosely the model answers, through the temperature of the reply. A strong personality gets a low temperature and sticks to its profile; a weak one gets a high temperature and wanders.

**What is temperature?**

When a language model picks its next word, temperature controls how adventurous it is. Near 0 it nearly always takes the most likely word, so it is steady and repetitive. Near 1 it will take less likely words too, so it is more varied and more likely to wander off.

### Personality strength to temperature

A panel shows the memories retrieved for each agent on its last turn. The profiles still tell agents to write [MOVE] when they want to leave a conversation, a leftover from an earlier version of the demo. In the grid demo, agents move through the movement tool instead.

## Numbers

The system has not been evaluated, so there is no accuracy figure to give. These are the fixed settings it runs with:

**Constants in the code, not measurements.**

| Setting | Value |
|---|---|
| Memory queries per turn | up to 3 |
| Hits per query | 3 |
| Memories put in the prompt | 9 at most, 5 in the newest version |
| Hybrid search alpha | 0.5 |
| Reply retries on bad XML | 3 (newest version only) |
| Default local model | qwen2.5:14b (Ollama) |

## What I ran into

**Retrieval ran against the mock, not the real database.** When I ran it on 11 March 2025, Weaviate answered with an HTTP 401 and the code fell back to the in-memory mock. My Docker Compose file starts only a Weaviate service, with no embedding service behind its transformers module.

**In the grid demo the memories reach the side panel, not the model.** James, Jade, Jesse and Jamal take their whole prompt from a short profile, and those profiles have no slot for memories, no tool list and no XML format. They are told to answer in two sentences. The retrieved memories show up in the side panel but are not handed to the characters.

**Ranking uses importance, not match quality.** After three searches, results are sorted by the importance number the model gave itself when it wrote the memory. A closer match loses to a higher-rated one.

**The newest version needs a cleanup before it starts.** One file, subconscious_agent.py, has duplicated docstrings and method stubs left over from an edit, and Python can't parse it. Another, agent_manager.py, has an await inside a plain function and never imports asyncio, which Python rejects at import. The earlier oli version, from 11 April, does not have that line.

**A few names and ids don't line up.** In the newest one, SubconsciousAgent renames its collection with capitalize(), so Memories_James becomes Memoriesjames, while AgentManager still creates Memories_James. Memories are also deduplicated on an id that results from a real Weaviate don't carry, and anything without one is dropped. So even once it parses, a real database would hand back nothing until that is fixed. The mock does carry the id, which is why it looked fine with the mock.

**There are no tests.** A memory is written every turn with no merging, and the merge_memories helper is not called anywhere.

## Where it stands

**The parts I planned, and where each stands.**

| Part | State |
|---|---|
| Get and make memories | Built |
| Edit memories (forget, rewrite) | Not started |
| Opinions from memory (oli version) | Teammate's stub, returns an empty string |
| Agent messaging over a websocket | Not started |
| 3D environment | Teammates' side, not part of what I published; the 2D grid stands in for it |

- **Stack**: Python, Weaviate (v4 client, Docker Compose), Ollama, Anthropic and OpenAI SDKs, Dash and Cytoscape
- **Status**: Prototype. Feb to Apr 2025. Not evaluated; the newest version does not start
- **Credit**: I led a team of four. The memory system, the agent loop and the grid demo in this repository are my side. Teammates worked on the environment and the 3D side and on some of the code; a teammate started the opinions part. Their work is not part of what I published and may be in their own projects.
- **Code**: [github.com/Forkei/Simu-Verse_Memory](https://github.com/Forkei/Simu-Verse_Memory) (my part only). The grid demo and the latest work are in the oli and oli2 versions of the project.

[← All projects](https://oliviercouthaud.com/projects)