Source: https://oliviercouthaud.com/jarvius · Status: Prototype · 2025 · Tags: Python, Discord, Whisper, Gemini, Weaviate

# Jarvius

My first AI assistant: a Discord bot you can type to or talk to, and the direct ancestor of what became VOS.

Jarvius began as a Discord bot in Python, using discord.py. It ran a handful of commands, sent everything else to a language model, could speak its replies in a voice call, remembered what each person had said, and ran a party game. Jarvius is the direct ancestor of VOS Gen 1: the same project, with the Discord bot as its first stage, from February to September 2025.

## What I wanted

I wanted Jarvis from the Iron Man films: an assistant you talk to out loud, that answers back and remembers you. The way I actually used it was on the walk to class. I would send it voice messages and ask it to reply in text or in voice, depending on whether I could listen or had to read. That worked. Discord was the shortcut. It already gives you accounts, text channels and voice calls, and a bot can sit in a call, so there was no interface to build before testing the idea. Every answer goes through Ollama, which runs open models on a machine you control. For me that was my own laptop, so the bot was not up around the clock, but it cost nothing to run. The first model was llama3.2:1b. By the end the default was dolphin3.

**What is Ollama?**

A program that downloads open language models and runs them on a machine you control, behind a small web address. The bot sends it a list of messages and gets text back, which is the same job a paid API does, without the bill. The cost is that you need a computer that can run the model.

When I created the project I described it as using a team of AI agents. The Discord version is one model behind one short instruction; the agents belong to the Gemini version that came after.

### How it grew

Six stops from the first text bot to VOS.

- Text bot. A Discord bot that answers prefix commands: ping, help, and two joke ones. Anything it doesn't recognise goes to a small model, llama3.2:1b, running in Ollama on a separate machine, with an instruction to keep answers to two sentences.
- Voice. On the same first day I wrote the voice side: join a voice channel, transcribe speech with Whisper and answer out loud with Parler-TTS. "hey jarbius" became the phrase that meant you were talking to it.
- Memory. It started keeping each person's conversation, so follow-ups worked, and a reply to one of its messages counted as talking to it without a prefix. The default model ended up as dolphin3.
- Death By AI. A separate party-game bot where the model narrates what happens to each player and decides who survives.
- VOS Gen 1. A Python assistant on Google's Gemini API, with several AI agents handing work to each other and a Weaviate database for memory. The direct successor of the Discord bot, and the same project. It cost more than I had planned. It taught me about APIs, cloud models and tracking usage, and I gave the agents a tool to check how much they had spent.
- VOS. The next generation, which I built with Roman Slack from September 2025. It has its own page.

Orange marks the current stop.

## How it works

There are three ways to get its attention: start a message with "hey jarbius", reply to one of its messages, or use a command prefix. If the first word is a known command it runs that directly: ping, help, memory, join, leave, generate, and the two joke ones, fart and poop, which print up to 100 random synonyms. Anything else goes to the model. The bot saves your message, builds a prompt from a short system instruction plus your recent history, sends it to Ollama and posts the answer. In the last version these normal chat replies run on a joke persona I swapped in, not on the Jarbius instruction. The Jarbius instruction is still what generate uses. Both ask for two sentences at most.

### Follow one message through the bot

Two routes through the same diagram. The typed route is the simple one. The spoken route is the call-listening loop, which the last version of the code no longer starts.

**Typed route**

- You write "hey jarbius", or reply to one of its messages, in a text channel. The bot sees it.
- The bot checks the first word. If it is a command such as ping, help or fart, it runs it and replies. The model is not involved.
- Otherwise it files your message under your server and your name in memory, and pulls back what you have said to it so far, up to 200 messages.
- It sends a short system instruction plus that history to Ollama. The instruction asks for two sentences at most. Ollama answers with text.
- The bot saves the reply to memory too, then posts it in the channel.
- If the bot is sitting in a voice channel, the reply is also turned into speech by Parler-TTS and played into the call.

**Spoken route**

- The bot sits in the call and records everyone in it as 48 kHz stereo audio (each person's audio is joined end to end).
- About every tenth of a second the loop measures how loud the recording is so far. If that is above 300 it counts as speech; after three seconds with no new reading above 300 the clip is closed. A clip is also cut at 30 seconds. Anything shorter than a second is thrown away.
- The clip is saved as a temporary audio file and handed to Whisper.
- Whisper returns a transcript. If "hey jarbius" appears anywhere in it, whatever follows is the request. Otherwise the clip is ignored.
- From here it is the typed route: a command runs directly, anything else goes through memory and Ollama.
- The reply is posted as text, with what it heard, and spoken into the call through Parler-TTS.

Orange boxes and arrows are the current step. The volume line of 300, the 3 second pause, the 30 second cap and the 1 second minimum are the settings the bot used.

Speaking back uses Parler-TTS mini v1. It takes the reply text plus a plain sentence describing the voice, and mine asks for a clear speaker on a close, clean recording. The result is played into the call through ffmpeg. If the bot is already talking, a new reply cuts the old one off. The listening side records the whole call, measures volume, and hands finished clips to Whisper.

**What is Whisper?**

An open speech-recognition model from OpenAI. You give it an audio file and it returns the text. The medium.en version is a mid-sized model trained on English only. It is more accurate than the small ones and slower, which matters when you run it on every clip.

**What does the volume check actually measure?**

It is the root mean square, or RMS, of the audio samples: square each sample, average them, take the square root. That turns a burst of sound into one loudness number. A quiet room gives a small one and speech a large one. The audio here is 16-bit, so single samples can reach about 32,000, and I set the line between quiet and speech at 300.

The wake phrase check is a plain text match on "hey jarbius" in the transcript, so a transcript that spells the name any other way is ignored. The last version of the code no longer starts the in-call listener: the bot joins your call, says so, and tells you to use generate to make it speak.

## Memory

Memory is a small class that keeps the last 200 messages for each person on each server, and a second log of the last 200 for the server as a whole. Each entry is a role, the text and a time. When the model is called, the prompt is a short system instruction plus that one person's history. That is why a follow-up like "and what about the second one?" works, and also why the bot has no idea what anyone else in the channel just said. The server-wide log is only counted by the memory command, which can show counts or clear one person or everything.

Figure: Diagram. Where a message is stored and which part of it the model gets to see. Orange is what goes into the prompt.

All of it sits in RAM, so a restart wipes it. The later Gemini version replaced it with Weaviate, which is a database that can keep memories between runs.

**What is Weaviate?**

A vector database. It stores each piece of text as a long list of numbers that captures roughly what it means, so a search can match on meaning and not just exact words. It also keeps everything on disk, so what it has stored survives a restart.

## Death By AI

The game is a separate program in the same project, with its own commands and no shared code with the assistant. It uses the same model on the same Ollama machine. One person creates a game in a channel, others join, and the creator starts it. It works with a single player too. There is one game per channel, and it closes itself after ten minutes of silence.

Figure: Diagram. One round. The orange box is the only place the model is called.

A random player is asked for a scenario and gets 30 seconds. If they don't write one, the bot picks one of twenty built-in scenarios. Everyone then has 60 seconds to say what they do, and anyone who doesn't answer dies. Each answer goes to dolphin3 together with the scenario. The instruction asks for a small block of XML: a short reflection on the scenario, the player's approach and the story, then a story of at most three sentences, then a verdict of SURVIVED or DIED with a one-line reason. The narrator is told to be a dark, crude comedian. The story is revealed by editing one message word by word, and a scoreboard follows each round.

**Game settings the creator can change**

| Setting | Default | Allowed |
|---|---|---|
| Rounds | 5 | a whole number |
| Time to write a scenario | 30 s | 10 to 120 s |
| Time to answer | 60 s | 10 to 300 s |
| Story reveal | word | character, word or instant |
| Delay between reveals | 0.2 s | 0.05 to 1 s |

A small model doesn't always answer in the format it was asked for, so the parser is forgiving. It adds the missing wrapper tag, escapes stray ampersands, and if it still can't read the answer, the player survives. A model error counts as survival as well. So a broken answer never kills anyone.

## What I ran into

Speech worked, with a delay: the speech models are loaded from scratch for every reply and for every clip, rather than once when the bot starts, which is the first thing I would change. I also rewrote the call-listening code a few times as I changed how the bot receives audio.

Memory lives in RAM, so a restart clears it, and it follows one person's thread at a time, not a group conversation. Everything depends on the Ollama machine being on, which was my laptop, so the bot answered only while it was on. The wake phrase is an exact text match, as above. Moving to Gemini later removed the laptop dependency and brought a usage cost to manage.

## Where it stands

The Discord bot is the first stage of the project. The work went on as VOS Gen 1 (see the timeline above) and then [VOS](https://oliviercouthaud.com/vos), built with Roman Slack. If I picked the bot up again I would load the speech models once at start-up so replies come back faster.

- **Stack**: Python, discord.py, Ollama on my own laptop (llama3.2:1b, later dolphin3), Whisper medium.en, Parler-TTS mini v1. VOS Gen 1: Gemini API, Weaviate.
- **Status**: Prototype
- **Period**: February to September 2025 (the Discord bot is the first stage of VOS Gen 1)
- **Code**: Not public
- **Why**: An assistant I could talk to on the walk to class
- **Next stage**: [VOS and VibeOS](https://oliviercouthaud.com/vos), built with Roman Slack