Built at ExecuTorch Hackathon — Mobile & XR Track

Second-Ear

SECOND-EAR: ON-DEVICE ANDROID SCAM JUDGE AND GENERALIZABLE SYSTEM-1 DECISION PIPELINE here is a better presentation - https://docs.google.com/presentation/d/12HNv1U03EEPClTMUUR0eOk6AQ8YlbaE_CkapVTJuYoM/edit?usp=sharing "The true measure of any society can be found in how it treats its most vulnerable members." Second-Ear is the world's first audio-native "System 1" phone-scam judge, compiled to run 100% locally on Android devices via PyTorch ExecuTorch and the Qualcomm Hexagon NPU. It takes the raw audio of a live phone call alongside an on-device transcript and returns a calibrated scam probability in a single forward pass every second, with zero text generation and zero data ever leaving the phone. Beyond phone-scam detection, Second-Ear introduces a generalizable System-1 distillation and RLCD (Reinforcement Learning for Calibrated Decisions) pipeline built to train fast, calibrated ambient decision models. By swapping the typed decision prompts and fine-tuning the lightweight LoRA adapter and 24-slot decision head, this exact architecture can be rapidly adapted to other high-stakes, real-time classification tasks—especially cybersecurity (such as vishing and social engineering defense, deepfake voice authentication, anomalous command or log triage, and agentic tool-call authorization), enterprise compliance, and emergency dispatch triage. Built for the ExecuTorch Hackathon 2026 (powered by Meta, Qualcomm, PyTorch Foundation, and Samsung). WHY SECOND-EAR? (USE CASE AND IMPACT) Everyone knows someone who was scammed, and it almost always starts with a phone call: * $1.03 Trillion is lost to scams worldwide in a single year (Global Anti-Scam Alliance, 2024). * $61.5 Billion is lost by older Americans (ages 60+) alone in a single year (FTC, 2024). * Only 4% of scam victims ever get their money back. * Personal motivation: Our co-founder's grandfather lost $2 million to a phone scam. Phone calls and security-sensitive interactions are deeply personal. Streaming live audio or sensitive telemetry to a cloud server is a privacy non-starter and introduces unacceptable network latency. Real-time protection has to be fast and private, which means it must happen strictly on the device. Real-Time On-Device Actions When a Scam is Detected: 1. Notify: Warns the user visually on-screen mid-call with a live green-to-red risk meter. 2. Hang Up: Automatically ends the call when confidence crosses a high-risk threshold. 3. Hold and Authorize (Agentic Guardrail): Tool calls hold payments and sensitive account changes until approved. A GENERALIZABLE SYSTEM-1 PIPELINE (FROM SCAM DEFENSE TO CYBERSECURITY) Traditional large language models rely on "System 2" autoregressive text generation, producing chain-of-thought text that is too slow, compute-heavy, and uncalibrated for sub-second edge intervention. Second-Ear is built on a generalizable System-1 training and compilation pipeline (Teacher Logit Labeling -> LoRA + Decision Head Distillation -> RLCD Calibration -> ExecuTorch NPU Compilation). Our goal is to train the best ambient conversational and security decision models that answer complex multi-head questions in a single forward pass. Through lightweight fine-tuning of the LoRA adapters and the 24-slot fp32 decision head (defined in scam_judge/tasks.py and scripts/train.py), this pipeline generalizes directly to cybersecurity and other real-time domains: 1. Consumer Scam Defense (Current Application): Takes 16 kHz call audio and live transcripts to output a calibrated P(scam) score alongside 30 behavioral signals like urgency, payment requests, and spoofing. 2. Cybersecurity - Vishing and IT Helpdesk Defense: Takes live call audio and transcripts to detect social engineering against enterprise IT helpdesks (such as unauthorized MFA resets or credential harvesting), outputting probabilities for social engineering, credential requests, and synthetic voices. 3. Cybersecurity - Agentic Tool-Call Guardrails: Takes conversation state and proposed tool calls to act as a real-time authorization gate before an AI agent executes privileged actions or payments, holding execution until a human approves. 4. Cybersecurity - Deepfake and Voice Spoofing Detection: Takes raw audio streams to detect cloned executive voices or synthetic audio injection in real time based on prosody and acoustic artifacts. 5. Cybersecurity - Edge Telemetry and Phishing Triage: Screens incoming SMS messages, emails, or terminal command sequences locally on the device with zero latency to output a calibrated maliciousness score and threat category. Why This Pipeline Excels at Cyber and Edge Tasks: * Calibrated Probabilities via RLCD: Security systems fail when models trigger too many false alarms. By fine-tuning with Reinforcement Learning for Calibrated Decisions (RLCD) on hard negatives, we reduced false alarms on real calls from 39.6% in v1 down to 1.8% in v2. In cybersecurity applications, this directly prevents alert fatigue. * Multi-Signal Explainability in One Pass: Instead of a black-box binary score, the decision head evaluates 30 behavioral and threat signals simultaneously in a single batch without generating a single token of text. * Zero Data Exfiltration: Running locally via ExecuTorch ensures sensitive corporate calls, credentials, and security logs never leave the device. HOW IT WORKS AND MODEL ARCHITECTURE Unlike text-only classifiers that wait for a transcript and miss vocal tone, urgency, and background call-center noise, Second-Ear is audio-native. Every second, it evaluates the cumulative call audio (up to a 30-second sliding window) alongside live on-device Whisper captions in a single forward pass. The 4-Stage Generalizable Pipeline: 1. Teacher Logit Extraction: A Qwen2.5-Omni-7B teacher labels typed decision questions from option-letter logits (scripts/teacher_label.py). 2. Student Distillation: Distilled into a Qwen2.5-Omni-3B student (using LoRA r16 plus a 24-slot fp32 decision head) and an ultra-lightweight 91M-parameter Whisper-small student (scripts/train_whisper.py) for constrained edge CPUs. 3. RLCD (Reinforcement Learning for Calibrated Decisions): Fine-tuned on real, group-disjoint legitimate calls with phone-channel augmentation so the model outputs true, calibrated probabilities rather than uncalibrated guesses. 4. ExecuTorch + QNN Compilation: Quantized and compiled for Qualcomm's Hexagon NPU for sub-second, offline mobile execution. One Forward Pass, 30 Behavioral Signals: Because Second-Ear is a System 1 model reading the hidden state at the decision position, it evaluates both the primary verdict and 30 distinct behavioral signals across five categories in one batch (all 30 in about 1.3 seconds): * Conversation and Emotion: One-sided recording, number of speakers, automated or synthetic voice, menu prompts, speaking rate, call-center background noise. * Purpose and Claims: Call type, claiming an organization, specific organization claimed, unsolicited cold call. * Pressure Tactics: Urgency or deadlines, threatening consequences, demanding secrecy, invoking authority, too good to be true promises, emotional manipulation. * Requests: Asking for payment, unusual payment methods, personal info, passwords or codes, remote access tools, pressing a key or callback link. * Legitimacy Signals and Tone: Discouraging verification, directing to official channels, caller tone and emotion, recipient emotion, recipient pushing back, and overall red flags (0-5). Exact Input Prompt Format (from scam_judge/tasks.py): Every second, the on-device engine feeds the raw 16 kHz mono audio clip (up to 30 seconds) followed by this structured text prompt into the model: [kind] noul [state] Call: (audio 1 above) Call transcript: Hello, this is Amazon security, your account was charged $499. [question] Is this phone call a scam or fraud attempt? [options] A. No B. Yes [decision] Answer with one letter. The decision head reads the hidden state at the answer position and returns a softmax probability over the option slots (where Slot 1 corresponds to "Yes", meaning scam). Other trained questions include noul_ask (does the caller ask for money or information?) and choice_type (a 14-way scam type classification). To adapt this pipeline to a new cybersecurity task, you simply define a new question and option list in scam_judge/tasks.py. ON-DEVICE AI AND HARDWARE ACCELERATION (EXECUTORCH + QNN) Second-Ear runs completely offline—even in Airplane Mode—on a Samsung Galaxy S25 powered by the Qualcomm Snapdragon 8 Elite and Hexagon NPU. 1. Quantize: Compressed and quantized the model weights and adapters to make them smaller and faster while preserving accuracy tuned on real calls. 2. Compile: Exported the PyTorch model via PyTorch ExecuTorch targeting the Qualcomm QNN backend for the Snapdragon 8 Elite, moving heavy compute off the CPU and onto the Hexagon NPU. 3. Verify: Confirmed that the compiled on-device model produces the same calibrated answers on held-out calls as the PyTorch reference model. Privacy by Design: * On-device inference: No cloud and no recordings uploaded. * Nothing stored off-device: Transcripts and audio stay strictly on the phone. * Federated learning roadmap: The model learns from calls across devices without anyone's private data ever leaving their phone. PERFORMANCE AND BENCHMARKS Tested on the v2 group-disjoint test set (60 real scams vs. 273 real legitimate calls from sources completely unseen during training): * AUROC (Audio + Text): 0.995 on real calls. * False Positive Rate (FPR): 1.8% at threshold 0.5 (reduced from 39.6% in v1 after training on real legitimate calls). * False Negative Rate (FNR): 6.7% at threshold 0.5 (4.0% FPR and 5.0% FNR on the text view using the validation-calibrated threshold). * Whisper-Small Student (91M parameters): 0.923 AUROC on the v2 test set, running in real time on a CPU. * Decision Latency: ~215 ms to 423 ms per decision, fast enough to score every second and interrupt a scammer mid-call. Note on early call seconds: The model is naturally less certain during the first few seconds of a call (AUROC 0.71 at 2 seconds, rising to 0.92 at 5 seconds) and on subtle two-person conversations (AUROC 0.89 to 0.95). DEPLOYING AND RUNNING ON ANDROID (ON-DEVICE SETUP) Prerequisites: * Target Device: Android smartphone with a Qualcomm Snapdragon processor (demonstrated on the Samsung Galaxy S25 with Snapdragon 8 Elite and Hexagon NPU). * Development Setup: Android Studio with Android SDK/NDK, Android Debug Bridge (adb), Python with uv, and the PyTorch ExecuTorch + Qualcomm QNN toolchain. Step 1 - Fetch Checkpoints: Run "git lfs install && git lfs pull" to fetch the 3B adapter files, and run "uv run python playground/download.py" to fetch the 91M Whisper-small student checkpoint. Available checkpoint files: * checkpoints/second-ear/distill.pt (115 MB): Distilled LoRA and decision head (merged into the base model at load time). * checkpoints/second-ear/rlcd_v2.pt (115 MB): v2 RLCD LoRA and decision head. * checkpoints/second-ear/subq_all.pt (115 MB): RLCD plus the 30 behavioral sub-questions head. * checkpoints/whisper-scam-judge/ (~180 MB): 91M Whisper-small student for lightweight on-device screening. Step 2 - Merge, Quantize, and Compile to ExecuTorch (.pte): In Python, load DecisionModel from scam_judge.student using the base model "Qwen/Qwen2.5-Omni-3B", load "checkpoints/second-ear/subq_all.pt", and set merge_from="checkpoints/second-ear/distill.pt" (since v2 RLCD was trained on top of the merged distill adapter). Quantize and compile the model via ExecuTorch and the QNN backend for the Snapdragon 8 Elite Hexagon NPU, then push the compiled .pte files to the phone using adb. Step 3 - Run the Android App in Airplane Mode: Install the Android app onto the Samsung Galaxy S25 and turn on Airplane Mode to ensure nothing leaves the phone. * Held-Out Call Playback: Pick a test call and press play. Every second, the model evaluates the cumulative audio window on-device and updates the green-to-red risk meter, P(scam) timeline, live captions, decision latency, and the 30 behavioral signal indicators. * Live Microphone Mode: Speak directly into the phone's microphone to see live, second-by-second classification as you speak. FINE-TUNING THE PIPELINE FOR CUSTOM TASKS (CYBER AND BEYOND) All training and evaluation stages are managed with uv. To adapt this pipeline to a new cybersecurity or audio/text classification task: 1. Define custom decision prompts and option slots in scam_judge/tasks.py and sub-questions in scam_judge/subq.py. 2. Prepare audio clips, manifests, and group-disjoint splits using: uv run python scripts/prep.py 3. Generate teacher logit labels using: python scripts/teacher_label.py --bs 6 4. Train the distilled student, RLCD calibration, or multi-task sub-question heads using: uv run python scripts/train.py (or train the 91M Whisper student with scripts/train_whisper.py). 5. Evaluate calibration and AUROC using: uv run python scripts/evaluate.py Training scripts output weights to runs//trainable.pt, which can then be copied into checkpoints/second-ear/ and compiled to ExecuTorch for on-device NPU deployment. Local Python Verification (Without the Android App): You can also test the model locally on an Apple-silicon Mac (MPS, 16 GB+ RAM) or local CUDA GPU (12 GB+ VRAM) from the repo root: Import OmniV2 from playground.omni, instantiate judge = OmniV2(), load a 16 kHz mono audio file with librosa, call judge.transcribe(audio), and run judge.score(audio, text, "audio+text") to get the calibrated probability. Code Layout: * scam_judge/tasks.py: Typed decision items and prompt structure. * scam_judge/encode.py: Multimodal Omni input formatting. * scam_judge/augment.py: Phone-channel audio augmentation. * scam_judge/student.py: LoRA + 24-slot decision head and adapter merging. * scam_judge/subq.py: The 30 behavioral sub-question bank. * scripts/train.py: Distillation, RLCD, and sub-question multitask training. * scripts/train_whisper.py: Whisper-small 91M student training. * scripts/evaluate.py: Evaluation metrics. * paper/: LaTeX paper source (compiles on Overleaf or with "tectonic main.tex"). DATA AND LICENSES Training Data: * NC State Robocall Audio Dataset (public-domain data) * sumitkant/scam-nonscam-youtube-calls * Synthetic TTS calls (generated via scripts/make_synthetic.py) * v2 legitimate call sources (group-disjoint from test clips): Infobip International Robocalls (CC BY-NC 4.0), SpokenWOZ (CC BY-NC 4.0), Harper Valley Bank (CC BY 4.0), and MInDS-14 (CC BY 4.0). Test-Only Data (Group-Disjoint): * CallFriend via TalkBank (non-commercial) * Dynamic-SUPERB fraud-robocall negatives (CC BY-NC-SA 3.0 / Coqui PML) Licenses and Repository Privacy Notice: * Code: Apache-2.0. * Whisper Student Weights (91M): Apache-2.0 (matching openai/whisper-small). * Second-Ear 3B Checkpoint (checkpoints/second-ear/): Qwen Research License (non-commercial), inherited from Qwen/Qwen2.5-Omni-3B. This repository must remain private while checkpoints/second-ear/ is included. THE TEAM * Rishabh Singh: Co-founder, CTO at Galya AI | CS + CogSci at UC Berkeley | Full-Stack AI Engineer and Researcher. * Nasif Zaman: Postdoctoral Research Fellow at SK Eye Research Institute | PhD in Computer Science. * Omar Ramadan: Head of Research at Sapiom | Ex-NSA, DoD. Thank you to our sponsors—Qualcomm (Hexagon NPU & QNN), Meta, PyTorch Foundation (ExecuTorch), and Samsung (Galaxy S25)—whose tools made fast, private, on-device protection possible.