Focus Buddy

What it watches for

It also tracks gaze and whether you are at the desk at all, both from geometry. Nudges are rate-limited - once a minute by default - and counted as episodes, not frames: a thirty-second phone check is one distraction, not thirty. Once an hour it tells you how the day is going. No score, no dashboard, no upload; the tally resets at midnight.

What it says

Spoken nudges come from a small set of fixed phrasings - several per habit, so a repeat offender does not hear the same line four times running. That is the default because it is instant, free, and never says anything strange to you.

A language model can rewrite them if you turn it on (FOCUS_BUDDY_USE_LLM_NUDGES): a 4-bit Llama-3.2-1B running locally through MLX on the edge-vlm path, or a hosted model otherwise. Whatever it writes has to pass validation before it reaches a speaker - right length, actually mentions the habit that was seen, and does not contradict the frame. Anything that fails is replaced by the fixed phrasing. A buddy that is confidently wrong about you is worse than one that repeats itself.

The hourly summary is not written by a model at all. "You've been working 2h, nail-biting 3 times" is formatted straight from the counters, because those numbers are facts about your day and should not be paraphrased by anything that can hallucinate. The voice is OpenAI text-to-speech, or the macOS say command, which is offline and free like the rest of the default path.

See it work

The phone comes out, the Mini notices, and says so. Nothing in that loop leaves the machine unless you ask it to.

Try it

Install it from the Reachy Mini dashboard, or run it against a laptop webcam:

pip install -e .
focus-buddy-setup-sidecar          # builds the MediaPipe environment, once
focus-buddy --desktop --backend edge

That needs no API key and no network. For phone detection and posture, set an OPENAI_API_KEY and use --backend cloud - from the dashboard, the app has a settings page for it. edge-vlm runs the old local VLM path and is kept for comparison, not recommended; see the table above for why.

Why geometry beat every vision model I tried

This is the finding the project is really about. "Is a hand touching your face" reads like a job for a vision-language model. It isn't - it's a distance, and the small local VLMs answer it at chance.

Measured on 16 labelled frames from the robot's own camera: eight with a hand at the face, eight without. Every 2-3B checkpoint answered yes to all sixteen.

The same frames separate cleanly by distance: 0.000-0.015 face widths from the face box while touching, 0.528-2.102 while not. A 35× gap, where the models had none. Thresholds are ratios of the face width, so they do not care how close you sit or what resolution the camera runs at. Gaze scored 23/24 from iris position; at-the-desk 41/43.

The models were not merely wrong, they were not looking. SmolVLM2 returned a character-identical sentence - "there is a hand touching the face, specifically near the mouth" - for three different frames and for a picture of coloured rectangles.

How it works

Every vision backend returns the same typed Observation, so swapping geometry for a cloud model changes nothing downstream. The four backend families - perception, brain, speech, hardware - each sit behind a Protocol built by a small factory from Settings, which is what lets the whole loop be tested against fakes with no camera, no network and no models.

Two rules the design does not bend on

The model never gets the last word. Nudges come from fixed phrasings by default. A language model may rewrite them, but whatever it produces must pass validation first - right length, actually mentions the habit it saw, and does not contradict the frame. Small models love to append "…and you're not even looking at your screen", which is often flatly untrue. Anything that fails is replaced by the template.

The summary is computed, not generated. "You've been working 2h, nail-biting 3 times" is formatted from counters. Those numbers are facts about your day and should not be paraphrased by anything that can hallucinate.

Getting a VLM onto an 8 GB Mac

Before geometry won, a lot of work went into making local vision models fit. That work is why the comparison above is fair - the models were given every advantage the hardware allows.

The machine is an M3 with 8 GB of unified memory, shared between CPU and GPU. SmolVLM2-2.2B at float32 wants about 8.8 GB, which does not fit; bfloat16 halves it to ~4.4 GB, which technically fits and then thrashes.

Why the usual 4-bit path does not work on a Mac

4-bit weights are packed on disk and unpacked to 16-bit floats on the fly, which needs a specialised kernel. In the Hugging Face ecosystem that kernel is bitsandbytes, written in CUDA - Nvidia's proprietary language. Apple GPUs use Metal, so the code cannot compile or run; it falls back to CPU or crashes. Apple's answer is MLX, written against Metal from the ground up, and mlx-vlm ships its own 4-bit kernels for Apple Silicon. That is the route this project took.

Bigger and quantized beats smaller and pristine

4-bit quantization keeps roughly 95-98% of a model's accuracy while removing 75% of its memory; 3-bit starts to degrade and 2-bit tends toward gibberish. Dropping to a 500M model instead lowers the ceiling structurally - it loses the capacity for spatial reasoning and stops respecting output formats. For "are the fingers at the lips or just near the jaw", a 4-bit 2.2B beats a pristine 500M. Confirmed here: the 500M checkpoint hallucinated hands onto the monitor.

What thrashing looks like

During inference the model touches essentially all of its weights for every token. If the working set exceeds RAM, the machine spends its time moving pages between disk and memory instead of computing - and disk is ~100× slower than RAM, so a task that should take seconds takes hours. The fingerprint is unmistakable: low CPU, tiny resident set, and swap climbing into the gigabytes. At its worst this build showed ~10 GB of swap in use on an 8 GB machine. The higher-precision SmolVLM2 convert took 339 seconds per frame before being abandoned.

Diagnostics that earned their keep: vm_stat 1 for live paging, sysctl vm.swapusage for the trend, memory_pressure for the kernel's own verdict, and /usr/bin/time -l for peak footprint and whether anything swapped at all.

Making a small model behave

Getting usable output from a 2B model is its own discipline. A cloud-style prompt asking for "looking_at_screen": True | … made SmolVLM echo the template back, placeholders included. A filled-in example made it emit the trailing fragment of the example. A numbered Yes/No checklist produced a stuck prior - the same answers regardless of the image.

What worked was to stop asking the model for structure at all: ask for one or two plain sentences about hands, mouth and gaze, then let Python turn that prose into the typed Observation. The format is the deterministic part, so it should not be spent on the model's limited capability. Every regex in that classifier exists because a real model said something the previous version got wrong - including the one that stopped "no indication of nail-biting" from registering as nail-biting.

The same lesson, taken further, is what produced the geometry backend: move work out of the prompt and into code wherever the work is mechanical. Distance is mechanical.

On the robot

The Mini's camera is wide-angle and its view is not a webcam's, so frames are cropped around the face before anything looks at them - biased downward toward hands and desk, and toward whichever way the head is turned. Face detection uses OpenCV Haar cascades rather than MediaPipe Tasks, which aborts in DrishtiMetalHelper on Apple Silicon. The head sways in time with speech by hooking the SDK's audio pipeline, which is cheaper and better-looking than a motion loop.

MediaPipe itself runs in a separate interpreter. Its only working API on Apple Silicon needs numpy<2, while reachy-mini needs numpy>=2.2.5 - there is no overlap, so it gets its own virtualenv and speaks over a pipe. The sidecar returns raw landmarks and the parent applies every threshold, so the geometry has exactly one tested copy.