GestureCam
A phone camera and a Python program that turn my body and hands into Minecraft controls. Prototype.
I wanted to play Minecraft with my body instead of a keyboard. Lean to one side and the character steps that way. Swing an arm and it swings a sword. Pull back an imaginary bowstring and it draws a bow.
I should say up front that it never got to the point where I could really play that way. The camera, the body tracking and the link into the game all work. What I never had was enough recorded examples to teach the network well, and I'll explain that below.
It has two halves. One is an Android app I wrote that turns a phone into a camera and sends the video to my PC. The other is a Python program on the PC. It finds the points on my body and hands in every frame, a bit like a stick figure, and a small neural network, which is a program that learns patterns from examples, looks at the last second of movement and decides what I'm doing. Then it presses the matching key in the game.
Two things I did not hand to the network. Strafing is a plain rule: if my shoulders sit about 4 centimetres off to one side of my hips, step that way. Looking around uses my left hand like a joystick. A fist, then an open palm, and where the palm sits turns the camera.
To teach the network I recorded around 300 short clips, most of them of one move each and some of me doing nothing. The first version learned nine moves. It was too eager to decide I was walking.
The second version skips the list of moves and learns the keys directly. When I stand still, it leaves the keyboard alone about 99 times out of 100. On the actual actions it lands around the middle of the scale, nowhere near what I'd call working. And the longer I trained it, the better it got on the clips it had practised on and the worse on clips it had never seen. It was memorising my clips instead of learning what the moves look like.
The labels were a shortcut too. I made them by converting my gesture clips, so a swing became an attack and walking became forward, instead of recording real play. It never saw an actual game.
What's next. For now it's on hold. Teaching it properly takes far more recorded examples than I had, and I got discouraged by how much I would still need to record. I recorded some, just not enough. I'd happily come back to it. The plan for that is in a separate project, MCGestureControl: play Minecraft normally and record it, then watch the recording and act it out in front of the camera, so every moment pairs my pose with the keys that were really pressed. The bigger model for that has not been trained.
What I wanted
I wanted to play Minecraft with my body instead of a keyboard, using a phone as the camera. Lean to strafe, swing an arm to attack, pull back an imaginary bowstring to shoot.
It never got to play properly. The tracking and the game link work, but the models I trained were not good enough to use, and the reason is mostly data. Teaching a network this takes a lot of recorded examples. I recorded some, not enough, and seeing how much more it needed put me off. I'd like to come back to it.
It grew in steps. On 19 February 2026 it was only an Android app that streams the camera to a PC, with manual control of exposure, ISO, focus, white balance and zoom, plus a Python client that tracked my hands. Pinch zoomed out, spread zoomed in, and a swipe up or down moved the exposure compensation by two steps. The next day I added body pose and a gesture classifier. By the 21st it was sending controls to Minecraft, and on the 22nd I started recording gameplay to train something better.
How it works
The app is Kotlin on Camera2. It captures JPEG frames and sends them over TCP on port 5000, each one preceded by a 4-byte length. A second socket on port 5001 takes JSON commands for the camera: exposure time, ISO, focus distance, white balance, zoom, resolution, JPEG quality and frame rate. On the PC, a client thread keeps only the newest frame, so a slow stage downstream drops frames instead of building up lag. On startup it lets auto-exposure and focus settle for a few seconds and then locks exposure, ISO, focus and white balance, so the picture does not shift while I move around. Pose and hand tracking run on the PC, not on the phone.
Step 1 of 9. The phone listens on port 5001 for small JSON commands: exposure, ISO, focus, white balance, zoom, resolution, JPEG quality, frame rate.
MediaPipe gives 33 body landmarks in world coordinates, in metres and centred on the hips. WiLoR-mini gives 21 3D keypoints per hand. Add a presence flag for each hand and a visibility score for every body landmark and that is 260 numbers a frame, plus 225 frame-to-frame velocities of the position values: 485 in all. The control policy also takes 16 values from the game and its own last five outputs, which makes 551.
The first model is a small Transformer encoder: 3 layers, 4 heads, width 128. It reads a window of 30 frames, about a second at 30 frames per second, and scores nine gestures: crouching, draw bow, jump, place block, sprinting, swing, sword draw, throw and walking.
What is a Transformer here?
The same family of network as the chat models, but small. It looks at all 30 frames of a window at once and learns which moments matter for which others. That suits a swing, which is a hand going back and then forward over about half a second.
Each gesture gets its own sigmoid output instead of one softmax over all of them, so walking and crouching can both be on. The thresholds are per gesture: 0.3 for walking, draw bow, place block and sword draw, which should fire easily, and 0.9 for sprinting and throw, which should not. In the game, walking and sprinting hold the forward key, sprinting also holds sprint, crouching holds sneak and drawing the bow holds right-click. Jump, swing and throw are one-off presses with a half-second cooldown. Sword draw only selects the first hotbar slot. I never wrote the part that finds the sword in the inventory.
Strafing and looking are hand-written rules, not learned. The strafe detector compares the sideways position of the middle of my shoulders with the middle of my hips, in metres from the pose tracker. A running average smooths the reading, and past 4 cm it presses the strafe key. It lets go at 3 cm.
Why two different thresholds?
That is hysteresis. If one number both switched the key on and off, a lean sitting right on 4 cm would flicker the key on and off every frame. With 4 cm to switch on and 3 cm to switch off, I have to come back a full centimetre before it lets go.
Looking around is a virtual joystick on my left hand. A fist arms it, opening the palm starts it, and from then on how far the palm sits from where it opened becomes yaw and pitch, with a small dead zone in the middle.
The second model, the control policy, skips the list of gestures. It reads the same window with a bigger Transformer (4 layers, 6 heads, width 192) and outputs eight on/off controls (forward, strafe left, strafe right, sprint, sneak, jump, attack, use item) and two look axes. None of my nine gesture classes is a strafe, so in the gesture path strafing still comes from the lean rule.
Results
I only have training numbers for the control policy. It trained for 45 epochs. By my combined score the best was epoch 20, with an average F1 over the eight controls of 0.479 and an idle accuracy of 0.992. By epoch 45 the F1 was 0.398 and the validation loss had gone from 1.00 to 7.02.
What do F1 and idle accuracy mean?
F1 runs from 0 to 1 and punishes both missed presses and false ones, so a control that never fires scores 0 and one that fires on every frame scores low too. I average it over the eight controls. Idle accuracy is separate: of the frames where I was doing nothing, how many it correctly left alone.
Picking a separate threshold for each control afterwards lifts the average F1 to about 0.52. I quote the 0.48 because that is the score before that tuning.
What didn't work
The policy memorised my clips. Training loss fell to about 0.012 (0.017 by epoch 45) while validation loss swung between 0.2 and 8.6.
What is overfitting?
A network can fit its practice data so closely that it learns the quirks of those exact clips instead of the pattern behind them. The giveaway is the top panel of the chart: error on the practice clips keeps falling, while error on clips it has not seen stops improving and then gets worse.
The control labels were not real play. I converted the gesture clips. Walking became forward, sprinting became forward plus sprint, a swing or a throw became an attack pressed at the moment my hand moved fastest, and drawing the bow became holding use item. I left out sword draw, because whether you want a sword depends on what is in your inventory, which the camera cannot see. In total that is 310 clips for ten outputs, which is not much. Four of the eight controls hit the ceiling of 50 on the positive-class weight. Strafe left and strafe right have no positive frames at all, because none of my clips is a strafe, and jump and attack only have a short pulse of a few frames per clip.
The first model had its own problems. Walking fired too easily at its 0.3 threshold. A gesture also sends the same key whatever is going on in the game, so drawing the bow does the same thing whether or not I'm holding a bow. That is why I started the second model, and why it takes game state as an input.
What's unfinished
A bigger policy for the next version is written: 671 inputs, 28 controls, about 3.5 million parameters and five output heads, with inventory and hotbar among them. It runs a forward pass without errors, but nothing has been trained. I recorded nine sessions of my own Minecraft play on 22 February to train it on, and that was not enough, so there is no result. It's on hold. I'd happily pick it up again, and the plan for it lives in MCGestureControl.
I budgeted roughly 23 to 26 ms per frame against the 33 ms you get at 30 frames per second. That is an estimate from adding up what each stage should cost, not a measurement.
Credits
The game side runs on MCCTP, a Fabric mod by Lucas Øyen that exposes Minecraft over a WebSocket. I wrote the Android app and the Python client. Pose tracking is MediaPipe and hand tracking is WiLoR-mini.
- Stack
- Kotlin and Camera2 on the phone. Python, PyTorch, MediaPipe and WiLoR-mini on the PC. Minecraft with Fabric.
- Status
- Prototype
- Started
- 19 February 2026