Project brief ExecuTorch Hackathon — Founders, Inc., October 17–18, 2026. Everything below runs on the Arduino VENTUNO Q on the desk: no cloud, no account, no network needed once the board is flashed. The pitch A store fixture that puts the shopper in the product photo. Stand in front of a camera and you appear wearing the garment — the head cutout, the body matte, the garment placement and the generated try-on image are all computed on the board's Hexagon NPU, with the web page that shows the result served from the same device. What's built and measured Piece Where it runs Cost Virtual billboard (BiSeNet head matte → product photo) NPU 12 ms compute, ~1 fps end to end Person segmentation (SegFormer human parsing) NPU 24.9 ms Generated try-on (Stable Diffusion + ControlNet-Canny) NPU ~0.3 s per step Face swap into a studio shot (GHOST, Apache-2.0) NPU 82 ms, fp16 LLM text generation (Gemma 4 E2B) NPU 188 tok/s prompt, 16.3 tok/s gen, 494 MiB Vision Q&A (InternVL3-1B) NPU 0.5 s encode, ~33 tok/s Live speech (Whisper-tiny → Gemma translate) NPU 1.8–2.6 s transcript, ~9 s translated LLM baseline (Llama 3.2 1B, XNNPACK) CPU 10.1 tok/s The NPU numbers are the point: prompt eval is ~12× the CPU baseline on a model twice the size, with resident memory a fraction of it, because the weights live in the NPU context. How it fits together Hugging Face model → optimum-cli export executorch → quantized .pte → ExecuTorch runtime. The same .pte runs on the Mac and on the board, which is what makes iterating on a laptop and demoing on hardware the same workflow. NPU models go through the track bundle's QNN AOT path (Linux only) and are driven by a warm persistent runner — 468 ms per inference launched cold, 26 ms held open with scratch on tmpfs. Two independent routes to the same effect, kept separate on purpose: overlay moves a garment onto a live frame (garment_overlay.py, garment_clip.py), and face swap moves the shopper's face into a studio shot that already has the garment, lighting and pose (board_faceswap_*). Hardware QCS8275, Cortex-A78C big cores, 14 GB RAM, Ubuntu 24.04 aarch64 over adb — not Android, despite answering adb, and considerably better than its published spec. No root; the compiler, the runners and the Python environment were all installed without it. Status and limits Board path: working, demoed, including the web UI and the walkthrough deck. Android/XR app (android/): compiles, never run — no device was attached. macOS cannot do QNN ahead-of-time compilation, so NPU work happens on the board. Thin segmentation classes (arms) lose accuracy at 8a8w; silhouettes don't. The board reboots under heavy load, so long board-side jobs are resumable.
Built at ExecuTorch Hackathon — IoT Track
Magic Mirror
Project brief ExecuTorch Hackathon — Founders, Inc., October 17–18, 2026. Everything below runs on the Arduino VENTUNO Q on the desk: no cloud, no account, no network needed once the board is flashed. The pitch A store fixture that puts the shopper in the product photo. Stand in front of a camera and you appear wearing the garment — the head cutout, the body matte, the garment placement and the generated try-on image are all computed on the board's Hexagon NPU, with the web page that shows the result served from the same device. What's built and measured Piece Where it runs Cost Virtual billboard (BiSeNet head matte → product photo) NPU 12 ms compute, ~1 fps end to end Person segmentation (SegFormer human parsing) NPU 24.9 ms Generated try-on (Stable Diffusion + ControlNet-Canny) NPU ~0.3 s per step Face swap into a studio shot (GHOST, Apache-2.0) NPU 82 ms, fp16 LLM text generation (Gemma 4 E2B) NPU 188 tok/s prompt, 16.3 tok/s gen, 494 MiB Vision Q&A (InternVL3-1B) NPU 0.5 s encode, ~33 tok/s Live speech (Whisper-tiny → Gemma translate) NPU 1.8–2.6 s transcript, ~9 s translated LLM baseline (Llama 3.2 1B, XNNPACK) CPU 10.1 tok/s The NPU numbers are the point: prompt eval is ~12× the CPU baseline on a model twice the size, with resident memory a fraction of it, because the weights live in the NPU context. How it fits together Hugging Face model → optimum-cli export executorch → quantized .pte → ExecuTorch runtime. The same .pte runs on the Mac and on the board, which is what makes iterating on a laptop and demoing on hardware the same workflow. NPU models go through the track bundle's QNN AOT path (Linux only) and are driven by a warm persistent runner — 468 ms per inference launched cold, 26 ms held open with scratch on tmpfs. Two independent routes to the same effect, kept separate on purpose: overlay moves a garment onto a live frame (garment_overlay.py, garment_clip.py), and face swap moves the shopper's face into a studio shot that already has the garment, lighting and pose (board_faceswap_*). Hardware QCS8275, Cortex-A78C big cores, 14 GB RAM, Ubuntu 24.04 aarch64 over adb — not Android, despite answering adb, and considerably better than its published spec. No root; the compiler, the runners and the Python environment were all installed without it. Status and limits Board path: working, demoed, including the web UI and the walkthrough deck. Android/XR app (android/): compiles, never run — no device was attached. macOS cannot do QNN ahead-of-time compilation, so NPU work happens on the board. Thin segmentation classes (arms) lose accuracy at 8a8w; silhouettes don't. The board reboots under heavy load, so long board-side jobs are resumable.
Keep exploring what builders shipped.
Harshith Ande's team
Kamikaze
Kamikaze is an edge-AI vision system built on the Arduino VENTUNO Q that converts physical consumer items into interactive 3D digital twins in real time. Powered by an on-device Qualcomm Hexagon NPU, the pipeline uses YOLO-World, DINOv2, RapidOCR, and HSV color gating to identify smartphones and beverage cans during a guided five-second scan. An integrated FastMCP server then resolves metric-accurate 3D models by cross-referencing local CAD caches, 1:1 dummy repositories, and procedural mesh generators with live-stitched label textures—all while enforcing a zero-cloud privacy boundary. The resulting Smart Kiosk interface pairs Modulino RGB LED feedback and real-time hardware telemetry with an auto-orienting 3D viewport that dynamically mirrors the physical pose of the object in the user's hand.
Team AAA
Depth Perception
Collating Arduino distance sensor readings with video to train camera-alone depth perception
BetweenTheLines
IDK Button - The button that knows what is being said
## Runtime Architecture: Board and Mac ASR and LLM inference run on the board using the Qualcomm NPU. The dashboard, application backend, and dictionary lookup logic run on the Mac. | Component | Execution location | Implementation | |---|---|---| | ASR | Board | Whisper compiled with ExecuTorch and executed through the QNN backend | | Context analysis and translation inference | Board | `unsloth/Qwen3.5-2B-GGUF` with Q4_0 quantization, served through GenieX using a llama.cpp-based inference engine and the Qualcomm NPU (QAIRT) compute backend | | Dashboard and application backend | Mac | Coordinate snapshots, context resolution, translation requests, and word selection | | Dictionary lookup | Mac | Reviewed dictionaries, OEWN 2025, and the project glossary | ASR and LLM inference do not require internet access. The complete application spans the board and the Mac: dictionary lookup and dashboard/backend processing are performed on the Mac. ## Whisper Execution with ExecuTorch and the Hexagon NPU ### Ahead-of-Time Compilation Whisper is compiled ahead of time with ExecuTorch. The compiled artifact, `whisper_qnn_fp16.pte`, is loaded using `_load_for_executorch()`. ### QNN Backend and NPU Execution The runtime loads `libqnn_executorch_backend.so` and explicitly asserts that `QnnBackend` is available. When `run_live_mic.py` starts the ASR process, it configures: ```text ADSP_LIBRARY_PATH=.../qairt/htp-files ``` This configuration supplies the Qualcomm Hexagon runtime libraries used by the QNN/QAIRT HTP backend. In the described deployment, Whisper encoder and decoder inference execute on the Hexagon NPU rather than through CPU fallback. Host-side orchestration and token selection remain separate from NPU model execution. ### Token-Level Decoding `live_board_audio.py` implements manual token-level greedy decoding and branch decoding through repeated calls to: ```python run_method('decoder', ...) ``` The decoding logic also computes no-speech probability and tracks 2-best branches. This logic produces the `asr_alternatives` and `branch_position` fields. ## Dictionary Lookup: Three Stages `live_board_audio.py` does not perform dictionary lookup. After ASR, it uses `urllib.request` to POST the recognized text to the configured backend endpoint: ```text http://127.0.0.1:8000/interpret/button ``` The URL above is preserved as configured; it should not be used on its own to infer the physical location of the backend. The dashboard/backend and dictionary logic described below run on the Mac, while Qwen inference runs on the board. ### 1. Before Translation: Ambiguity Resolution Small, reviewed regex-based dictionaries help resolve ambiguous words and phrases before translation: - `context_catalog.SENSES`: senses for `ground`, `port`, `bank`, and `fixed` - `phrasal_lexicon.PHRASES`: expressions such as `make it` Relevant call flow: ```text dashboard_server.process_snapshot() → /context/resolve → context_resolution.resolve_context() ``` `dashboard_server.process_snapshot()` calls `/context/resolve` as soon as a snapshot is frozen. These dictionaries provide deterministic rules for known ambiguities. Ambiguities not covered by the rules fall back to Qwen-based sampling. ### 2. Immediately After Translation: “Words worth keeping” After translation, the dashboard searches the full OEWN (Open English Wordnet) 2025 dataset and the project glossary, stored in SQLite, to suggest words worth saving. Call flow: ```text dashboard_server.translate_snapshot() → dictionary_options() → /dictionary/suggestions → dictionary_suggestions.suggest_words() → dictionary_store.search_dictionary() ``` This lookup supports vocabulary collection after translation. ### 3. During Word Selection: Knob-Based Word Exploration When the user turns the knob to explore words, the system first checks the small, reviewed dictionaries: - `word_meaning.GLOSSES` - `word_meaning.FUNCTION_KO` If no matching entry exists, it requests a meaning from Qwen running on the board. Call flow: ```text schedule_word() → /api/words/meaning → word_meaning.resolve_word() ``` The three lookup stages serve separate purposes: resolving ambiguity before translation, suggesting vocabulary immediately after translation, and explaining individual words during user-driven exploration. Polished Demo : https://drive.google.com/file/d/1-jO-gHfr4oEMIjxGgMtZJLsF9clInU3c/view?usp=sharing