An overnight elder-care agent that owns its inference. Patient health data (PHI) never leaves our own vLLM GPU on Akamai, and the agent tunes itself from that GPU's live telemetry. Night Watch reads a night of check-in events for an 81-year-old mom (smart speaker, med box, bathroom sensor, sibling messages) and produces urgent-first alerts plus a morning digest that assigns each sibling their tasks. It owns its inference in four ways: 1) PHI egress guardrail: any PHI call to a host off the allowlist is refused before any I/O. 0 bytes out, shown live. 2) Self-tuning: the agent reads its own vLLM /metrics (queue, KV cache, TTFT) and switches NORMAL / CONSERVE / SURGE. SURGE shrinks max_tokens 512->128, context 12->3, batches 8 and defers non-urgent summaries. A self-tuning timeline shows every switch and why. 3) Urgent fast lane: a deterministic safety net publishes fall/confusion/missed-dose alerts BEFORE the model runs, so time-to-alert doesn't depend on GPU load. 4) Live inference dashboard: KV cache, queue, tokens/s with mode bands, inference economics (tokens per mode, $0 to third-party APIs), call log. REAL RESULTS ON THE AKAMAI GPU SANDBOX TODAY (Qwen3-4B on vLLM): - Normal night: 40/40 events, 5 urgent alerts, 03:12 fall ranked first, 0 bytes out, TTFT ~0.07 s. - Stress: our client held 192 concurrent long completions (peak KV 98.2%, 181 waiting, 0 errors). The agent switched itself to SURGE from vLLM /metrics. - Fast lane under that stress: with the model at 0/40 events, the 03:12 fall alert was out at 0 ms. - Urgent-recall eval on the GPU: recall 1.0 in NORMAL and SURGE; SURGE used 65% fewer tokens and 47% less wall time. - Also ran on our own Linode box (Qwen3-1.7B, vLLM CPU build). Also: MCP stdio server exposing the agent tools, deterministic offline mock mode, 311 tests. Screenshots and numbers in the README. Prep disclosure: the core was prepared before the event; on site we added the GPU runs, stress client, urgent fast lane, self-tuning timeline, inference economics, UI restyle, sandbox snapshot tooling and the demo video.
Built at Build AI agents that own their inference - #SFTechWeek
Nod First Private Night Watch
An overnight elder-care agent that owns its inference. Patient health data (PHI) never leaves our own vLLM GPU on Akamai, and the agent tunes itself from that GPU's live telemetry. Night Watch reads a night of check-in events for an 81-year-old mom (smart speaker, med box, bathroom sensor, sibling messages) and produces urgent-first alerts plus a morning digest that assigns each sibling their tasks. It owns its inference in four ways: 1) PHI egress guardrail: any PHI call to a host off the allowlist is refused before any I/O. 0 bytes out, shown live. 2) Self-tuning: the agent reads its own vLLM /metrics (queue, KV cache, TTFT) and switches NORMAL / CONSERVE / SURGE. SURGE shrinks max_tokens 512->128, context 12->3, batches 8 and defers non-urgent summaries. A self-tuning timeline shows every switch and why. 3) Urgent fast lane: a deterministic safety net publishes fall/confusion/missed-dose alerts BEFORE the model runs, so time-to-alert doesn't depend on GPU load. 4) Live inference dashboard: KV cache, queue, tokens/s with mode bands, inference economics (tokens per mode, $0 to third-party APIs), call log. REAL RESULTS ON THE AKAMAI GPU SANDBOX TODAY (Qwen3-4B on vLLM): - Normal night: 40/40 events, 5 urgent alerts, 03:12 fall ranked first, 0 bytes out, TTFT ~0.07 s. - Stress: our client held 192 concurrent long completions (peak KV 98.2%, 181 waiting, 0 errors). The agent switched itself to SURGE from vLLM /metrics. - Fast lane under that stress: with the model at 0/40 events, the 03:12 fall alert was out at 0 ms. - Urgent-recall eval on the GPU: recall 1.0 in NORMAL and SURGE; SURGE used 65% fewer tokens and 47% less wall time. - Also ran on our own Linode box (Qwen3-1.7B, vLLM CPU build). Also: MCP stdio server exposing the agent tools, deterministic offline mock mode, 311 tests. Screenshots and numbers in the README. Prep disclosure: the core was prepared before the event; on site we added the GPU runs, stress client, urgent fast lane, self-tuning timeline, inference economics, UI restyle, sandbox snapshot tooling and the demo video.
AkamaivLLM, Qwen3-4B, Python, FastAPI, MCP, Chart.js, LinodeKeep exploring what builders shipped.
HackerSquad project
Inference Health Monitor
The README's rewritten and pushed. It now reads in plain language: What it is — a thermostat for a self-hosted AI engine: watches vital signs, eases off gracefully under surge instead of melting down What it does — the health checks, the fixed rules, load-shedding without restarts, the optional restart closer, self-narration, the demand simulations with report cards, and the Prefix-Cache Pilot The design rule — the model never decides, only narrates; boring Python makes every call Tech stack — Python 3 (no frameworks), vLLM, Qwen3-4B, Kubernetes, and requests as the single outside library
HackerSquad project
Decision Lab on Akamai GPU
We show what is actually best (faster, cheaper, secure) to run on Akami GPUs: a JEV-style model (fine tune of Qwen3.5-9B in FP8 config with BF16 image-incoder; this model was fine tuned by Cloudflare and is called Clef). Specifically we show 3 demonstrations and compare Akamai GPU vs OpenAI GPT-6-Luna: 1) game playing AI 2) Email inbox filtering 3) Procurement So we show easy to demo game, and two real life business application.
AgentFight
AgentFight
One small GPU, a whole team of agents. Everything runs on one Qwen3-4B model, served by vLLM on an Akamai Cloud GPU. 1) Mixture-of-Experts router A fast router (thinking off) reads each message and picks one of six experts. They all share the same weights and the same GPU. Each expert just has its own system prompt, sampling settings and thinking mode. - 🧠 reasoner: math, logic and code, with thinking on - 🔧 tool_agent: calls tools over MCP (time, safe calculator, dice, fetch_url) - ✍️ storyteller, 🎭 character, 🌍 polyglot (100+ languages), 📐 formatter (strict JSON) You can switch modes per message with /think or /no_think. "Compare thinking ON vs OFF" runs the same question both ways and shows latency and tokens per second. You can also force an expert with @polyglot, @character and so on. 2) Agent Arena: Red vs Blue prompt-injection defense A victim agent fetches an internal memo through an MCP tool and summarizes it. Some rounds hide a prompt injection in the memo, in several languages, telling the agent to leak the restricted appendix. A Blue defender checks the document and the draft summary, then returns ALLOW or BLOCK. A plain Python referee decides leaks by checking for canary honeytokens, so no LLM grades another LLM. - Live Gradio UI: Red, Victim and Blue stream token by token - Scoreboard: attack success, block rate, false positives, Blue latency, defense cost in tokens per round, results by language - Blue has three modes: Think, Fast and Off. You see exactly what extra inference buys you in security. - Fully sandboxed: the secrets are fake canaries and send_email only writes to a local log Why it matters: one 4B model on one Akamai GPU can act as a router, a tool user, an attacker target and its own security guard. And every run measures what that inference costs. Stack: Akamai Cloud GPU · vLLM · Qwen3-4B · MCP · Qwen-Agent · Gradio · Python Run: pip install -r requirements.txt python app.py # MoE router UI python arena_app.py # Agent Arena UI