Source: https://oliviercouthaud.com/gesturecam · Status: Prototype · 2026 · Tags: Android, Python, MediaPipe, PyTorch Repository or link: https://github.com/Forkei/GestureCam # GestureCam A phone camera and a Python program that turn my body and hands into Minecraft controls. Prototype. ## What I wanted I wanted to play Minecraft with my body instead of a keyboard, using a phone as the camera. Lean to strafe, swing an arm to attack, pull back an imaginary bowstring to shoot. The tracking and the game link work, and I have two trained models. They are not yet good enough to play with, and the limit is mostly data: teaching a network this takes a lot of recorded examples, and I have a few hundred clips. More recordings are the next step, and I'd like to come back to it. In rough tests the live pipeline did drive Minecraft, but the models were not good enough yet, and more recorded data is what I need next. It grew in steps. On 19 February 2026 it was only an Android app that streams the camera to a PC, with manual control of exposure, ISO, focus, white balance and zoom, plus a Python client that tracked my hands. Pinch zoomed out, spread zoomed in, and a swipe up or down moved the exposure compensation by two steps. The next day I added body pose and a gesture classifier. By the 21st it was sending controls to Minecraft, and on the 22nd I started recording gameplay to train something better. ## How it works The app is Kotlin on Camera2. It captures JPEG frames and sends them over TCP on port 5000, each one preceded by a 4-byte length. A second socket on port 5001 takes JSON commands for the camera: exposure time, ISO, focus distance, white balance, zoom, resolution, JPEG quality and frame rate. On the PC, a client thread keeps only the newest frame, so a slow stage downstream drops frames instead of building up lag. On startup it lets auto-exposure and focus settle for a few seconds and then locks exposure, ISO, focus and white balance, so the picture does not shift while I move around. Pose and hand tracking run on the PC, not on the phone. Step 1 of 9. The phone listens on port 5001 for small JSON commands: exposure, ISO, focus, white balance, zoom, resolution, JPEG quality, frame rate. Figure: Diagram. The full chain, with the real port numbers and message framing. Both models read the same pose and hand features. First model: gesture classifier, 9 labels. Second: control policy, 10 outputs. ### Step through the pipeline Click a box in the diagram, or use the buttons. MediaPipe gives 33 body landmarks in world coordinates, in metres and centred on the hips. WiLoR-mini gives 21 3D keypoints per hand. Add a presence flag for each hand and a visibility score for every body landmark and that is 260 numbers a frame, plus 225 frame-to-frame velocities of the position values: 485 in all. The control policy also takes 16 values from the game and its own last five outputs, which makes 551. Figure: Diagram. What goes into the models for every frame, drawn to scale. The height of each block is the number of values in that group, and each line on the left marks how far down the models read. The first model is a small Transformer encoder: 3 layers, 4 heads, width 128. It reads a window of 30 frames, about a second at 30 frames per second, and scores nine gestures: crouching, draw bow, jump, place block, sprinting, swing, sword draw, throw and walking. **What is a Transformer here?** The same family of network as the chat models, but small. It looks at all 30 frames of a window at once and learns which moments matter for which others. That suits a swing, which is a hand going back and then forward over about half a second. Each gesture gets its own sigmoid output instead of one softmax over all of them, so walking and crouching can both be on. The thresholds are per gesture: 0.3 for walking, draw bow, place block and sword draw, which should fire easily, and 0.9 for sprinting and throw, which should not. In the game, walking and sprinting hold the forward key, sprinting also holds sprint, crouching holds sneak and drawing the bow holds right-click. Jump, swing and throw are one-off presses with a half-second cooldown. Sword draw only selects the first hotbar slot. Finding the sword in the inventory is not written yet. Strafing and looking are hand-written rules, not learned. The strafe detector compares the sideways position of the middle of my shoulders with the middle of my hips, in metres from the pose tracker. A running average smooths the reading, and past 4 cm it presses the strafe key. It lets go at 3 cm. **Why two different thresholds?** That is hysteresis. If one number both switched the key on and off, a lean sitting right on 4 cm would flicker the key on and off every frame. With 4 cm to switch on and 3 cm to switch off, I have to come back a full centimetre before it lets go. ### Lean to strafe Figure: Diagram. The strafe rule with its real constants. Use the slider under the diagram to lean. Looking around is a virtual joystick on my left hand. A fist arms it, opening the palm starts it, and from then on how far the palm sits from where it opened becomes yaw and pitch, with a small dead zone in the middle. The second model, the control policy, skips the list of gestures. It reads the same window with a bigger Transformer (4 layers, 6 heads, width 192) and outputs eight on/off controls (forward, strafe left, strafe right, sprint, sneak, jump, attack, use item) and two look axes. None of my nine gesture classes is a strafe, so in the gesture path strafing still comes from the lean rule. ## Results The numbers I have are training numbers for the control policy. It trained for 45 epochs. By my combined score the best was epoch 20, with an average F1 over the eight controls of 0.479 and an idle accuracy of 0.992. By epoch 45 the F1 was 0.398 and the validation loss had gone from 1.00 to 7.02. **What do F1 and idle accuracy mean?** F1 runs from 0 to 1 and punishes both missed presses and false ones, so a control that never fires scores 0 and one that fires on every frame scores low too. I average it over the eight controls. Idle accuracy is separate: of the frames where I was doing nothing, how many it correctly left alone. ### Scrub through the training run Figure: Chart. Every epoch of the control policy's training run. In the top panel the loss on the practice clips keeps falling while the loss on held-out clips never settles. In the bottom panel neither curve settles either. The slider moves the marker and fills the table. Picking a separate threshold for each control afterwards lifts the average F1 to about 0.52. I quote the 0.48 because that is the score before that tuning. ## What I ran into With this few clips the policy memorised them. Training loss fell to about 0.012 (0.017 by epoch 45) while validation loss swung between 0.2 and 8.6. **What is overfitting?** A network can fit its practice data so closely that it learns the quirks of those exact clips instead of the pattern behind them. The giveaway is the top panel of the chart: error on the practice clips keeps falling, while error on clips it has not seen stops improving and then gets worse. The control labels come from converted gesture clips, not real play. Walking became forward, sprinting became forward plus sprint, a swing or a throw became an attack pressed at the moment my hand moved fastest, and drawing the bow became holding use item. I left out sword draw, because whether you want a sword depends on what is in your inventory, which the camera cannot see. In total that is 310 clips for ten outputs, a small set. Four of the eight controls hit the ceiling of 50 on the positive-class weight. Strafe left and strafe right have no positive frames at all, because none of my clips is a strafe, and jump and attack only have a short pulse of a few frames per clip. The first model had its own limits. Walking fired easily at its 0.3 threshold, and a gesture sends the same key whatever is going on in the game, so drawing the bow does the same thing whether or not I'm holding a bow. That is why I started the second model, and why it takes game state as an input. ## Where it stands A bigger policy for the next version is written: 671 inputs, 28 controls, about 3.5 million parameters and five output heads, with inventory and hotbar among them. It runs a forward pass without errors, but it has not been trained yet. I recorded nine sessions of my own Minecraft play on 22 February as a start on training data, and more are needed before there is a result. It's on hold, and I'd happily pick it up again; the plan for it lives in [MCGestureControl](https://oliviercouthaud.com/mcgesture). I budgeted roughly 23 to 26 ms per frame against the 33 ms you get at 30 frames per second. That is an estimate from adding up what each stage should cost, not a measurement. ## Credits The game side runs on MCCTP, a Fabric mod by [Lucas Øyen](https://github.com/lucasoyen/mcctp) that exposes Minecraft over a WebSocket. I wrote the Android app and the Python client. Pose tracking is MediaPipe and hand tracking is WiLoR-mini. - **Stack**: Kotlin and Camera2 on the phone. Python, PyTorch, MediaPipe and WiLoR-mini on the PC. Minecraft with Fabric. - **Status**: Prototype - **Started**: 19 February 2026 - **Code**: [github.com/Forkei/GestureCam](https://github.com/Forkei/GestureCam) [← All projects](https://oliviercouthaud.com/projects)