Source: https://oliviercouthaud.com/vos · Status: Prototype · 2025-26 · Tags: Python, Rust, Flutter, RabbitMQ # VOS and VibeOS An assistant built from a team of cooperating AI agents, and a smaller one that lives inside a virtual computer, both built with Roman Slack. [Image: Dark desktop-style interface with a notes window, a weather window, a calendar and a chat window, and a launcher rail on the left] Figure: The VOS app, designed by Roman. The notes, weather and chat shown are test data. ## Why I built it I wanted an assistant I could talk to rather than type at, closer to Jarvis in the Iron Man films than to a chat box. I built it with Roman Slack, and then I used it quite a bit myself. That turned out to be the useful part. Using it kept turning up things to fix, so I was probably improving it more than using it. That is how I'd work on every project from now on: use the thing early, because that is where the real problems show up. ## What it is VOS is the assistant. We built it between September 2025 and February 2026. Under the hood it is not one AI but a small team of them. One agent, the primary agent, is the only one that talks to you. When you ask for something it can't do itself, it hands the job to a specialist: one for weather, one for searching the web, one for your calendar, one for notes, one for maths, one for browsing websites. Think of a front desk with a few colleagues behind it. You ask the desk, the desk walks your question over to the right colleague, and brings the answer back to you. Figure: Diagram. The front desk and the colleagues behind it. ## What it can do You can type to it, talk to it, or phone it. It keeps a memory, so something you said last week can come up again without you repeating it. The window-style app at the top is Roman's design. I wrote the talking part, the phone screen and parts of the notes and settings screens. The first thing that worked from end to end was asking for the weather in New York and getting the answer back through three separate programs. After that it grew one specialist at a time. ## What I ran into Agents sometimes got stuck halfway through a job or handled the same message twice, and I spent a lot of time on that. On a phone call, while it is speaking it ignores you, so it doesn't hear its own voice and answer itself. The cost of that choice is that you can't interrupt it. The original plan was bigger than what exists: it included a security agent that watches the others, games, and a team of agents that writes code. Those are not built, and a few launcher icons, like the shop, are placeholders that open nothing yet. ## VibeOS VibeOS is the smaller sibling. Roman started it in late January 2026 as a minimal agent, written in Rust (a programming language). There is one agent, Vi, that lives inside a virtual computer and works the way you would at a keyboard: it types commands into a terminal. A second character, Joi, is the voice. You talk to Joi, and if what you asked for is a job she passes it to Vi, then tells you the result in her own words. The glowing orb screen is Roman's work. I wrote the part that decides what Vi does next, the lock on the door of the virtual computer and a small web dashboard. One bug I'm glad I fixed: the AI would sometimes describe the result of a command before the command had run, which means it made the answer up. Now it pauses mid-sentence, the real command runs, and it carries on from the real output. [Image: A glowing orb in orange, yellow and violet on a black screen above the word LISTENING, a question about the weather in Salt Lake City and a spoken-style reply, with VibeOS written at the bottom] Figure: The VibeOS voice app while it listens. Roman built this screen. Roman and I also worked on Rookery, a feed for research papers. I wrote the part that decides which papers to show you. The site is no longer online. ## What's next Both are prototypes and stay as they were in February 2026, which is when the last work on them was done. What I spend my time on now is a newer system of agents set up like a company. That one is not public. ## What I wanted An assistant I could talk to rather than a chat box: one agent owns the conversation, hands work to specialists, remembers what I said last week, and can take a phone call. I built VOS with Roman Slack from September 2025 to February 2026. The original plan was bigger than what exists. What follows is the part that is built. I also used it a fair amount myself, and that shaped most of the work. Using it kept turning up problems, so I probably spent more time improving it than using it. That is how I work on every project since: use it early and often, because the bugs you find that way are the ones that matter. I wrote the SDK every agent is built on, the API gateway, the primary agent and most of the specialists, the call system, the Twilio gateway, and an API for editing system prompts with versioning and rollback. Roman wrote the browser agent, backend auth, the memory visualisation, the first small web frontend and, late on, the call fast mode. Roman finished the notes agent, and the voice gateway is shared work. The Flutter client is Roman's repo: he designed it and wrote the window system, and I added voice mode, the notes UI, the phone app and the prompt settings. [Image: The same shell in October 2025: notes cards, a weather window for San Francisco, a calendar for October 2025 and a chat in which the assistant creates a welcome note] [Image: The Flutter shell in September 2025: a launcher rail on the left and four empty windows labelled Notes, Analytics, Chat and Calendar, with the text 'This is working' in the input bar] September 2025October 2025 Figure: The client, before and after. Drag to compare. Left, Roman's first shell with empty windows. Right, about six weeks later, connected to the agents. All content is test data. ## How VOS works Every agent is the same SDK class with a different tool list. A message from the client goes through the API gateway into RabbitMQ as a typed notification. The primary agent reads it and, if needed, sends an agent_message to a specialist's queue. The specialist runs its tools and reports back. In text chat only the primary agent talks to the user. In a phone call the call can be transferred to a specialist, which then speaks through the call tools. **What is RabbitMQ?** A message broker. Every agent has its own queue, which works like a mailbox. A sender drops a message in and moves on, and the agent picks it up when it is free. A message only leaves the queue once the agent confirms it handled it, so a crash in the middle doesn't lose it. Figure: Diagram. The services as they run in Docker. Agents read and write PostgreSQL and Weaviate directly through the SDK and its tools, so the lines to the databases show the dependency, not every call. Arrows show the direction a request travels. ### Follow one request The real path of a text question, "What's the weather in New York?", step by step. The weather itself is not shown, only the route. Use the buttons, or tap a box to jump to the first step it appears in. Figure: Diagram. The path of one request. - You send the question from the app. It goes to api_gateway. - api_gateway wraps it as a user_message notification and drops it into primary_agent's queue in RabbitMQ. - primary_agent wakes up, loads its history from PostgreSQL, looks up memories in Weaviate and asks Gemini what to do. The answer has to contain at least one tool call. Here it is send_agent_message, addressed to weather_agent. - That tool puts an agent_message into weather_agent's queue. - weather_agent calls get_weather. The tool fetches the data from OpenWeatherMap and posts a tool_result notification back to weather_agent's own queue. - weather_agent reads that result on its next turn and sends the answer to primary_agent as another agent_message. - primary_agent calls send_user_message. That goes to api_gateway over HTTP, and api_gateway pushes the reply to the app over a websocket. The loop inside each agent is a three-state machine kept in the database: IDLE, THINKING, EXECUTING_TOOLS. A turn loads history, calls Gemini in JSON mode, and requires at least one tool call. An empty list is rejected. Tools publish their results back to the agent's own queue as new notifications, so a multi-step task is several turns, not one long call. Figure: Diagram. One agent turn. The retry count of 3 and the 300 s limit are constants in the SDK. Memory has two modules, both set to run every turn by default (the interval is an environment variable). A retriever runs before the model call and a creator after the turn. The creator is itself an LLM call, and it answers CREATE_NOW, WAIT or IGNORE for what just happened. The retriever searches Weaviate and feeds the results into the turn. Memories are either private to one agent or shared across agents. Notes use the same store for semantic search. **What is semantic search, and what is Weaviate?** Weaviate is a vector database. It stores each piece of text as a long list of numbers that captures roughly what it means, so a search can match on meaning instead of exact words. Asking about "my trip next month" can find a note that only says "flight to Lisbon on the 14th". That is the whole trick behind both the agents' memories and the notes search. ### The team and what each agent can do - **primary_agent**: Talks to the user, delegates and summarises. The only agent that can message the user, phone them or send an SMS. send_user_message, send_agent_message, create_task, update_task, get_tasks, assign_to_task, unassign_from_task, sleep, create_memory, search_memory, get_memory, update_memory, delete_memory, create_document, read_document, list_documents, delete_document, get_attachment, view_image, read_system_prompt, edit_system_prompt, speak, answer_call, hang_up, transfer_call, recall_phone, call_user, call_phone, send_sms - **weather_agent**: Current weather and forecast through OpenWeatherMap. 5 domain tools. get_weather, get_forecast, get_weather_by_coordinates, get_uv_index, get_air_quality - **search_agent**: Web research through Firecrawl (scraping, crawling, extraction) and DuckDuckGo. 8 domain tools. firecrawl_scrape, firecrawl_crawl, firecrawl_map, firecrawl_search, firecrawl_extract, ddg_text_search, ddg_news_search, ddg_books_search - **calendar_agent**: Events and reminders, with a scheduler service behind it. 8 domain tools. create_calendar_event, list_calendar_events, update_calendar_event, delete_calendar_event, create_reminder, list_reminders, edit_reminder, delete_reminder - **notes_agent**: Notes, with semantic search. 8 domain tools. create_note, list_notes, get_note, update_note, delete_note, search_notes, archive_note, pin_note - **calculator_agent**: Arithmetic, statistics, linear algebra, number theory and units. 7 domain tools. basic_calculation, advanced_math, statistics, random_number, number_theory, linear_algebra, unit_conversion - **browser_agent**: Browser automation, written by Roman. 2 domain tools. browser_use, browser_navigate Every specialist also gets the shared tools: send_agent_message, task and memory tools, sleep and shutdown, and the call tools speak, hang_up and transfer_call for when a call is passed to it. ## Voice and calls Voice has two paths. Voice mode streams audio to voice_gateway, which transcribes with AssemblyAI (I moved over from Deepgram in November) and speaks with ElevenLabs or Cartesia, chosen per session. Call mode goes through a call bridge in the same gateway. It waits 1.2 s after the last transcript before sending text to the agent, and it drops transcripts while the agent's own speech is playing, to stop echo. Calls can be transferred between agents, each with its own voice. Phone calls arrive through twilio_gateway, which converts Twilio's 8 kHz mulaw audio to PCM and back. A fast mode, Roman's, uses a smaller model and allows only the speak and hang-up tools. **Why wait 1.2 seconds before answering?** Speech-to-text hands back a transcript in pieces, one each time you pause for breath. If every piece went straight to the agent it would answer half-sentences. So each new piece restarts a 1.2 s timer, and only when the timer runs out does the combined text go to the agent. It costs a little delay on every turn and it stops the agent jumping in early. ## VibeOS A Rust agent, vos_agent, runs inside a headless Ubuntu VM under QEMU and executes bash through a PTY. A gateway exposes it over WebSocket with an API key, rate limiting and TLS. Memory is SQLite with full-text search. Roman started it in late January 2026 as a minimal agent, with the simplest solutions and minimal code as the rule, and built most of the voice pipeline, the later plugin system, the memory rework and the WhatsApp and Twilio work. I wrote the agent loop, the TLS and rate-limited gateway, the first version of agent memory (remember, recall, forget), a multi-agent delegate tool and a web dashboard. The voice side is a separate agent, Joi (Python, Groq, Hume TTS). She listens, decides whether a request is a task for Vi, the Rust agent, and speaks the reply. In voice mode Vi never speaks to you directly; in the chat app it does. **What is a PTY?** A pseudo-terminal: a fake terminal window that a program can type into and read from. It lets the agent run bash the way a person at a keyboard would, with a real shell, working directory and interactive programs, instead of launching one command at a time with no shell around it. Figure: Diagram. The agent also writes to the same database, and OpenAI and Ollama are configurable providers. Arrows show the direction a request travels. My part of the loop went through three versions. First a loop capped at five iterations. Within days I replaced that with a 200-iteration safety valve that ends only when the model calls an explicit stop tool, rejects turns with no tool call, and turns every error into context for the next turn. The third version, in February, runs tools inline. The model writes one continuous stream, and when it writes a tool tag such as the stream pauses, the real command runs, and its output is spliced in before the model carries on. I made the change because the model had been inventing tool results before they ran. I also replaced regex detection of "is this a task" with a Groq model in Joi. [Image: A glowing orb in orange, yellow and violet on a black screen above the word LISTENING, a question about the weather in Salt Lake City and a spoken-style reply, with VibeOS written at the bottom] Figure: The VibeOS voice app while listening. Joi takes the question and Vi does the lookup. The orb is Roman's work. ## Results I have not benchmarked either system, so there are no latency, accuracy or cost numbers. The first thing that worked end to end in VOS was weather: ask the primary agent, it delegates to the weather agent, the answer comes back. After that the specialists, memory, voice and calls came one at a time. VOS has no automated tests. VibeOS has close to 190 small unit tests in Rust plus an integration test for the gateway. The limits I set in the code are these. **Limits I set in the code. Not measured behaviour.** | Setting | Value | |---|---| | VOS agent: LLM call timeout | 90 s | | VOS agent: requeue on transient error | 3 times | | VOS agent: stuck state reset | 300 s | | VOS call: transcript debounce | 1.2 s | | VibeOS: tool loop, iterations | 200 | | VibeOS: bash output kept, chars | 20,000 | ## What I ran into Agents got stuck or double-handled messages, and most of the reliability work went there: a fix for a race on message history in October, a requeue system in September and a reset for stuck agents in December. The memory creator saved the same thing over and over until I gave it a duplicate check and made it ignore when unsure. I changed models several times, from Gemini 2.5 Flash to a 3 Pro preview and on to Flash variants, without a side-by-side comparison, so I have no ranking of them. Call mode drops speech while the agent is talking, which stops echo; the trade-off is that you can't interrupt it. Roman's Twilio work in VibeOS was added untested and was later switched off. ## Where it stands Both projects stay as they were in February 2026. My current work is a newer system of agents organised like a company. It is not public. The original plan also had a Watchdog security agent, event subscriptions, a games and apps suite and a coding team of agents; those are not built. There is a registry service for apps, but nothing that generates them yet. Basic document tools exist (create, read, list, delete), and the agent-to-agent document flow I planned is the next piece there. The app's launcher has icons for a shop, reminders and tasks that are placeholders for now. In VibeOS, Joi's personality is not written yet, so in February Roman added a direct Vi mode to test voice without her. ## Also with Roman: Rookery Roman Slack and I also made Rookery, a social-media-style feed for research papers. I built the backend and the AI that decides which papers to show you; Roman built the interface and handled publicity. A waitlist was online; the site was not opened to the public. It builds paper embeddings from the citation graph with a GraphSAGE network, and the adaptive feed mixes mostly relevant papers with fresh ones and a share of deliberately adjacent topics. I evaluated it with simulated users. The site is no longer online. - **Stack**: VOS: Python, FastAPI, RabbitMQ, PostgreSQL, Weaviate, Gemini, Flutter. VibeOS: Rust (tokio, axum), SQLite, QEMU, Python for Joi. - **Status**: Prototype - **Built with**: Roman Slack - **Code**: VOS: [github.com/Forkei/VOS-public](https://github.com/Forkei/VOS-public), and Roman's [VOS_frontend](https://github.com/RomanSlack/VOS_frontend). VibeOS: not public. [← All projects](https://oliviercouthaud.com/projects)