Abstract
Live camera VLM systems are often described by model throughput, but their useful latency is a product of capture, transport, vision encoding, repeated text work, decoding and scheduling. We built a local-first lab around Glance and ran 21 preregistered experiments to separate those costs.
Native multi-question batching produced a 2.405× speedup. A smaller 2B model was 2.994× faster than the 4B reference but failed the quality guardrail. The accepted 8-bit MLX direct scorer reduced fresh-frame p50 from 358.5 to 259.6 ms (1.381×), matched 84/84 fixed-suite decisions and kept maximum probability drift to 0.039.
The main lesson is methodological: optimize realized model work, not configuration labels or proxy counts. Several attractive changes reduced nominal tokens or precision without producing a safe end-to-end win.
1. System and method
Glance answers typed questions about a frame by reading the probability of constrained answer tokens directly. It does not generate prose. The live system wraps that scorer with camera capture, a scheduler and a same-origin local gateway.
camera → crop / resize → local gateway → vision encoder
→ shared-prefix question batch → answer-token probabilities
→ scheduler / temporal policy → browser telemetryEach experiment declared a hypothesis, primary metric, quality guardrail, hardware, version and stopping rule before implementation. Accepted speedups had to survive paired measurement and preserve decisions within the declared tolerance. The fixed suite used nine statements across reusable test images; raw frames and model weights are excluded from the public artifact.
Apparatus. Apple M5, 32 GB unified memory. Reference: Qwen3-VL-2B-Instruct via Glance, PyTorch MPS FP16. Candidate: Qwen3-VL-2B-Instruct via MLX, 8-bit.
2. Results
| Intervention | Measured result | Decision |
|---|---|---|
| Native multi-question request | 2.405× faster; exact decisions within tolerance | Keep |
| 8-bit MLX direct scoring | 358.5 → 259.6 ms p50; 84/84 decisions; max drift 0.039 | Ship experimentally |
| 2B instead of 4B | 2.994× faster; 83.3% agreement | Fast tier only |
| 4-bit MLX direct scoring | Speed held; maximum probability drift reached 0.361 | Reject |
| Fixed-shape suffix compilation | 0.6% p50 improvement | Below materiality |
| Uniform vision-token reduction | Up to 15.6% faster; one decision changed, drift reached 0.411 | Reject |
| Untrained early decoder exits | 6–22% faster; no tested depth passed both gates | Reject |
| Browser payload work | Base64 and JSON cost at most 0.1 ms p95 at 320 px | Deprioritize |
The MLX result was not accepted on latency alone. It preserved every fixed-suite decision in the paired run, replicated at 1.334×, and was then integrated into Glance behind an explicit backend flag. PyTorch remains the compatibility default.
Native batching was the clearest architectural win: one shared visual prefix and one multi-question request avoided repeated setup. By contrast, browser serialization was measurable but too small to explain user-visible latency.
3. What did not work
- Quality fail4-bit quantization. It retained speed but moved probabilities too far for a dependable scorer.
- No material winCompilation. Fixed-shape suffix compilation improved p50 by only 0.6%.
- Quality failUniform token cuts. Lowering visual resolution removed useful evidence along with compute.
- Quality failRaw early exits. Shallower decoder states were faster, but were not trained to be calibrated decision heads.
- Wrong bottleneckTransport micro-optimization. At the tested frame size, model compute dominated the browser-to-loopback path.
Keeping these results matters. They narrow the search space and expose a repeated failure mode: a cheaper representation is not automatically a cheaper or equivalent computation.
4. Interpretation
The successful changes reduced work the hardware actually performed: shared-prefix batching removed duplicate model passes, while MLX paired an Apple-native runtime with accepted 8-bit weights and direct answer-token scoring. The rejected changes mostly optimized a proxy—declared precision, token count or graph form—without securing both realized latency and decision stability.
The evidence points toward conditional compute rather than one globally smaller model:
cheap temporal gate
↓ changed enough
fast 2B / MLX scorer
↓ uncertain or fine-detail
full reference pathThis cascade is still a hypothesis. The synthetic temporal gate reduced triggers by 95.1%, but requires labeled real-camera sequences before it can support a public efficiency claim.
5. Limitations
- The primary measurements come from one Apple M5 machine with 32 GB unified memory.
- The fixed suite is deliberately small and measures decision preservation, not broad semantic capability.
- Thermal state, other Apple chips and lower-memory machines were not characterized.
- The temporal result is synthetic; no real-camera quality or staleness study has been completed.
- This is a working technical report and open artifact, not peer-reviewed evidence.
6. Next hypotheses
- A trained intermediate decision head can retain full-depth margins while recovering the 11–22% compute exposed by raw truncation.
- Learned or task-conditioned vision-token selection can recover the measured prefix opportunity without uniform-resizing quality loss.
- Model-shaped fused 8-bit Metal kernels can improve the accepted MLX path beyond generic eager execution.
- A calibrated 2B→4B cascade can retain most of the 2B speedup while escalating uncertain or fine-detail frames.
- Task-aware temporal reuse can lower effective compute on labeled camera streams while bounding staleness.
7. Reproducibility and availability
The stable typed-question protocol, direct probability readout, shared-prefix batching and optional MLX backend live in Glance. Speedlab contains the live A/B interface, client telemetry, benchmark scripts, failed variants and full research record.