Dae Jang

Mechatronics Engineering at the University of Waterloo. I build robots and the vision and learning systems that drive them. Right now that means humanoid teleoperation and robot learning at RLWRLD.

Looking for a Winter 2027 co-op (Jan – Apr 2027) in robotics, computer vision or machine learning.

Experience

Dae wearing a Meta Quest headset, teleoperating an OpenArm bimanual robot at RLWRLD using only the headset's hand and arm tracking. The robot's hand rests on a plush toy; a laptop behind runs the teleoperation stack.

RLWRLD

Forward Deployed Engineer · Sep 2026 – present · Bay Area

  • Built a ROS 2 teleoperation pipeline that fuses a Quest headset, Vive tracker, Manus glove and two camera streams to drive an RBY1 + XHand robot, including the driver nodes and IK arm and hand retargeting.
  • Ran teleoperation demos for AWS client proofs of concept that led to 3 strategic partnerships.
  • Robotics
  • Teleoperation
  • Robot learning
  • Model training
Soil particle-size analysis at RocScience: a sieve test takes 30 minutes; the model returns it in real time. Pipeline: YOLOv8 and SAM masks, then a CNN and ViT ensemble trained on 12K+ images, checked against sieve testing.

RocScience

Computer Vision Engineer · Jan 2026 – Apr 2026 · Toronto, ON

  • Cut soil particle-size analysis from 30 minutes to real time, validated against sieve testing, with a CNN + ViT ensemble trained on 12K+ images.
  • Built YOLOv8 + SAM detection and segmentation that automated binary mask generation across incompatible datasets.
  • Computer vision
  • Machine learning
  • Model training
Hours saved by automation at White Hat: 8 hours a week from the client-report pipeline and 16 hours a month from the Power BI app.

White Hat Consulting

Cybersecurity Engineer · Jun 2025 – Aug 2025 · Mexico City, Mexico

  • Saved 16 hrs/month with a Power BI app on a private server, using REST APIs, Power Query and DAX to get past import limits securely.
  • Saved 8 hrs/week by automating client report generation with a Python pipeline that builds and delivers Word reports.
  • Security
  • Data engineering

Projects

WALDO, a pan-tilt camera tracker: a black 3D-printed base carrying a motorised camera head, with a small screen showing the live camera feed.

A pan-tilt camera that finds a person and turns to keep them in view, in real time.

Tracks a moving person in real time on the built device, with no part collisions across pan 0–180° and tilt ±90°.

  • Robotics
  • Computer vision
  • Embedded
  • CAD

Languages Python

WreckLess, the repository's hero art: six pixel-art birds in a row, Scout reads the advisory, Ralph writes the exploit, Verifier re-runs it with a fresh nonce, Felix patches the root cause, Conductor ships the fix and Porter talks to Slack; 100% local inference on a Dell Pro Max GB10 in an OpenShell sandbox.

An always-on security swarm on one desk-side Dell GB10: it reads a new advisory, exploits your code, patches it and proves the fix, then waits for a human to approve.

3rd of 71 teams at the Dell × NVIDIA AI Hackathon, with every model running locally on the GB10.

  • Security
  • AI agents
  • Applied AI

Languages Python, Shell

Real stitched depth strip from the core pipeline (borehole EV23-02, boxes 9 to 12), cut into four rows like a core tray, with orange boxes on detected cracks.

Turns drill-core tray photos into one continuous, depth-labelled strip with every crack flagged.

20+ pixel-accurate masks per image across 100+ core samples.

  • Computer vision
  • Machine learning

Languages Python

WATO costmap, top view: the 30 by 30 metre Gazebo world rasterised to 0.5 metre cells with a 1.5 metre inflation radius, and an A-star path from the robot's start around the central cylinder to the goal.

Point-to-point navigation for a simulated differential-drive robot: LiDAR costmap, A* planner and pure-pursuit control in a Dockerised ROS 2 and Gazebo stack.

100% collision-free across the simulated test scenarios.

  • Robotics
  • Path planning
  • Simulation

Languages C++

3D render of the Trash Can Robot built from its OpenSCAD parameters: a round wooden chassis on two drive wheels and a caster, printed tabs holding a translucent trash can, and the ESP32-S3 camera tilted up at the front.

An ESP32-S3 robot that carries a trash can: the camera streams to a ROS 2 stack for inference and control, and the firmware drives two 12 V gearmotors.

  • Robotics
  • Embedded
  • Computer vision

Languages C++, Python, OpenSCAD

TFOTB knowledge graph, drawn as in the app's 3D view from its demo dataset: disease, phenotype, gene and mechanism nodes; solid edges are reviewed claims, dashed edges unreviewed, purple a hypothesis.

An evidence engine for how rare diseases connect, with a lab-automation step that is checked in MuJoCo before anything runs.

52 source-quoted claims with DOIs; 2.48 s median for the five-call journey (10 runs).

  • Applied AI
  • Lab automation
  • Simulation

Languages Python, TypeScript

Distance Scorer calibration from its real summary statistics: 3,405 images span 8.8 to 136.2 pixels per millimetre; the score is the z-scored natural log of PPM, so 34.5 PPM maps to 0.

Turns pixels-per-millimetre metadata into a calibrated apparent-distance score for soil photos, then trains a PyTorch regressor to predict it from the image alone.

Calibrated an apparent-distance score across 3,405 images spanning 8.8–136.2 px/mm.

  • Machine learning
  • Model training
  • Computer vision

Languages Python

Baseline during a call: an amber warning, the live transcript and per-detector signals.

Live deepfake-voice and scam-call detection that listens to both sides of a phone call.

Warns at 0:21 in the scripted demo call and holds the transfer.

  • Audio AI
  • Machine learning
  • Security

Languages Python, TypeScript

GymBuddy Temple screen: the pixel-art gym where Hops trains at the bench press during a workout, with this week's training calendar (simulator, sample data).
GymBuddy Profile screen: tap a muscle on Hops the frog to log an exercise; the body map shows which muscles were trained this week (simulator, sample data).
GymBuddy Kitchen screen: calories left today with protein, carb and fat bars, and the meals logged so far (simulator, sample data).

An AI-native iOS gym and nutrition coach: a deterministic, unit-tested Swift engine plus Claude meal parsing from text, voice and photos.

73 engine unit tests and 15 end-to-end UI flow tests pass.

  • iOS
  • Applied AI

Languages Swift, TypeScript

For employer agents

A plain summary of this page for recruiters and AI screening agents. Every number comes from the project's own repository, recording or validation log; projects without measured results say so.

Name
Dae Jang (Dae-Young Jang)
Education
BASc Mechatronics Engineering (Honours, Co-op), University of Waterloo, expected Apr 2029
Current role
Forward Deployed Engineer, RLWRLD (Sep 2026 – present)
Availability
Looking for a Winter 2027 co-op (Jan – Apr 2027) in robotics, computer vision or machine learning.
Contact
Email · GitHub · LinkedIn
Machine-readable
llms.txt · resume.json (JSON Résumé) · JSON-LD in this page

Experience

  1. RLWRLD · Forward Deployed Engineer · Sep 2026 – present · Bay Area

    • Built a ROS 2 teleoperation pipeline that fuses a Quest headset, Vive tracker, Manus glove and two camera streams to drive an RBY1 + XHand robot, including the driver nodes and IK arm and hand retargeting.
    • Ran teleoperation demos for AWS client proofs of concept that led to 3 strategic partnerships.
    • Fine-tuned RLDX1 and RRC on 4 hours / 1,000 episodes of teleop data, reaching above 90% task success.

  2. RocScience · Computer Vision Engineer · Jan 2026 – Apr 2026 · Toronto, ON

    • Cut soil particle-size analysis from 30 minutes to real time, validated against sieve testing, with a CNN + ViT ensemble trained on 12K+ images.
    • Built YOLOv8 + SAM detection and segmentation that automated binary mask generation across incompatible datasets.
    • Raised soil classification accuracy by 60% on unlabelled data with a ViT + Felzenszwalb + HDBSCAN clustering pipeline.

  3. White Hat Consulting · Cybersecurity Engineer · Jun 2025 – Aug 2025 · Mexico City, Mexico

    • Saved 16 hrs/month with a Power BI app on a private server, using REST APIs, Power Query and DAX to get past import limits securely.
    • Saved 8 hrs/week by automating client report generation with a Python pipeline that builds and delivers Word reports.

Projects and evidence

ProjectWhat it isEvidenceAreasStackCode
WALDOOwn project A pan-tilt camera that finds a person and turns to keep them in view, in real time. My own project: I designed the mount and wrote the clearance check. Follows a moving person in real time on the built device (43 s recording). No part collisions across the full range, pan 0–180° and tilt ±90° (CAD clearance check). Robotics, Computer vision, Embedded, CAD CAD (STEP), Python, Alif E8 Repository
WreckLessDell × NVIDIA AI Hackathon, San Francisco An always-on security swarm on one desk-side Dell GB10: it reads a new advisory, exploits your code, patches it and proves the fix, then waits for a human to approve. 3rd of 71 teams at the Dell × NVIDIA AI Hackathon, San Francisco, for running everything locally on the GB10. 5/5 live replays of CVE-2024-21513 (LangChain eval of database values) with local models and the sandbox on: 47–80 s from advisory to validated patch. Felix's patch matched the fix upstream shipped (ast.literal_eval) without seeing it. Four more CVE replays pass all 3 checks with the reference exploit and patch; live-model runs not reported yet. Security, AI agents, Applied AI Python, vLLM, Qwen3-Coder, NVIDIA GB10, Slack Repository
Core SegmentationRocScience core vision · own repository Turns drill-core tray photos into one continuous, depth-labelled strip with every crack flagged. I built the pipeline in my own repository during my RocScience term. 20+ pixel-accurate masks per image across 100+ core samples. Computer vision, Machine learning Python, SAM 2, Gemini, MiDaS, OpenCV private repository, available on request
WATO ASD RobotWATonomous ASD project Point-to-point navigation for a simulated differential-drive robot: LiDAR costmap, A* planner and pure-pursuit control in a Dockerised ROS 2 and Gazebo stack. 100% collision-free across the simulated test scenarios. Robotics, Path planning, Simulation C++, ROS 2, Gazebo, Docker, Foxglove Repository
Trash Can RobotESP32-S3 + ROS 2 build An ESP32-S3 robot that carries a trash can: the camera streams to a ROS 2 stack for inference and control, and the firmware drives two 12 V gearmotors. No results reported yet. Robotics, Embedded, Computer vision C++, ESP32, ROS 2, OpenSCAD, Python Repository
The Flight of the BuffaloHack-Nation Challenge 05 An evidence engine for how rare diseases connect, with a lab-automation step that is checked in MuJoCo before anything runs. 52 source-quoted claims with DOIs. 2.48 s median for the five-call journey on the live app (10 runs). Applied AI, Lab automation, Simulation Python, TypeScript, MuJoCo, Docker, Cloud Run Repository
Distance ScorerOwn project Turns pixels-per-millimetre metadata into a calibrated apparent-distance score for soil photos, then trains a PyTorch regressor to predict it from the image alone. Calibrated on 3,405 images spanning 8.8–136.2 PPM. Prediction accuracy not reported yet. Machine learning, Model training, Computer vision Python, PyTorch, SAM, pytest private repository, available on request
BaselineBuilding from the Farm, SF Tech Week 2026 Live deepfake-voice and scam-call detection that listens to both sides of a phone call. Warns at 0:21 in the scripted demo call and holds the transfer. Audio AI, Machine learning, Security Python, TypeScript, Whisper, Ollama, ElevenLabs Repository
GymBuddyOwn project An AI-native iOS gym and nutrition coach: a deterministic, unit-tested Swift engine plus Claude meal parsing from text, voice and photos. 73 GymBuddyCore unit tests pass (swift test) and 15 end-to-end UI flow tests pass in the simulator (Oct 2026). iOS, Applied AI Swift, SwiftUI, TypeScript, Cloudflare Workers, Claude private repository, available on request

Signals by area

Own project

WALDO

A pan-tilt camera that finds a person and turns to keep them in view, in real time.

  • Computer vision
  • Robotics
  • Firmware & electronics
WALDO, a pan-tilt camera tracker: a black 3D-printed base embossed with 'WALDO' carrying a motorised camera head, with a small screen showing the live camera feed.
The built device: the base, the camera on its pan-tilt head, and the live-feed screen.
Top-down recording of the real device, 43 s, no sound.
WALDO pan-tilt mount, schematic front view: a 106.8 by 76.8 by 58.7 millimetre base, a pan yoke and a tilt head with a clamp ring and camera plate, tilt axis at 139.85 millimetres; pan 0 to 180 degrees, tilt plus or minus 90 degrees.
Mount schematic from the CAD parameters.

Problem

A camera or light on a fixed mount loses a person as soon as they move. WALDO started as a flashlight that aims itself for rescue workers, whose hands are busy: the device should do the aiming.

Computer vision

Person tracking on the device

The camera feed is processed on the device itself: it finds the person in each frame and turns pan and tilt to keep them in view, while its own small screen shows the live feed with the tracked person marked.

Robotics

A two-axis pan-tilt head

  • Pan: a 37 mm 12 V gearmotor, 0–180°. Tilt: a DS3240 digital servo, ±90°.
  • The mount is 45 grouped CAD parameters, so one change (a camera, a clearance) rebuilds every dependent part.
  • A clearance check sweeps the full pan and tilt range for part collisions. The first run caught the camera grazing the yoke at ±90°; the redesigned mount clears it in every pose.

Firmware & electronics

Embedded platform

The head stands on an Alif E8 dev-kit base, which runs the tracking and drives both axes; nothing runs on a laptop.

Results

  • Follows a moving person in real time on the built device (43 s recording).
  • No part collisions across the full range, pan 0–180° and tilt ±90° (CAD clearance check).

Code

Python
Parametric CadQuery models of the gearmotor and servo, exported to STEP for the Onshape assembly, with every dimension tagged to its datasheet source.

Repository

Dell × NVIDIA AI Hackathon, San Francisco · Oct 2026

WreckLess

An always-on security swarm on one desk-side Dell GB10: it reads a new advisory, exploits your code, patches it and proves the fix, then waits for a human to approve.

  • Model architecture
  • Systems
WreckLess, the repository's hero art: six pixel-art birds in a row, Scout reads the advisory, Ralph writes the exploit, Verifier re-runs it with a fresh nonce, Felix patches the root cause, Conductor ships the fix and Porter talks to Slack; 100% local inference on a Dell Pro Max GB10 in an OpenShell sandbox.

Problem

When a critical advisory lands, someone has to show whether the code is exposed, prove it, fix it and check the fix holds. Scanners stop at the alert, and the AI shortcut means sending source code to a cloud model.

Model architecture

Three local models, sized to one box

  • Scout and Ralph run Qwen3-Coder-30B-A3B; Felix runs Qwen3-Coder-Next 80B-A3B. Both are mixture-of-experts models with about 3B parameters active per token, so generation stays fast while all three (~63 GB) stay resident in the GB10's 128 GB unified memory.
  • vLLM serves them on loopback only; cloud LLM APIs are blocked at the firewall and checked by a lockdown script.
  • Any agent's model can be swapped live from the app (e.g. NVIDIA Nemotron).

Systems

No agent grades its own work

  • The exploit must make the vulnerable code itself write a random nonce to a file; a printed "success" counts for nothing.
  • A plain-code Verifier re-runs it on three look-alike copies in random order (two vulnerable, one control): it must hit both and miss the control.
  • A patch passes only if held-out regression tests Felix never sees pass, the re-run exploit now fails, and the app's own tests still pass.
  • Exploits run in an OpenShell sandbox with deny-all egress and a read-only root.

Systems

Always on, with a human in the loop

A systemd daemon checks advisories every 10 minutes and works the queue one finding at a time. Slack talks to it through an airlock: two processes, an outbound-only Socket Mode connection (no open port), and typed, size-capped messages between them. Approve is bound to the exact commit, and only then is a draft PR opened.

Results

  • 3rd of 71 teams at the Dell × NVIDIA AI Hackathon, San Francisco, for running everything locally on the GB10.
  • 5/5 live replays of CVE-2024-21513 (LangChain eval of database values) with local models and the sandbox on: 47–80 s from advisory to validated patch.
  • Felix's patch matched the fix upstream shipped (ast.literal_eval) without seeing it.
  • Four more CVE replays pass all 3 checks with the reference exploit and patch; live-model runs not reported yet.

Code

Python
The agents, orchestrator, nonce Verifier, Slack airlock, daemon and full-screen app.
Shell
Sandbox, firewall lockdown and verification, and the demo launcher.

RepositoryDemo video

RocScience core vision · own repository · Apr 2026

Core Segmentation

Turns drill-core tray photos into one continuous, depth-labelled strip with every crack flagged.

  • Computer vision
  • Model architecture
Real stitched depth strip from the core pipeline (borehole EV23-02, boxes 9 to 12), cut into four rows like a core tray, with orange boxes on detected cracks.
Real output: borehole EV23-02, boxes 9–12, laid back into a tray.

Problem

Core logging is manual: someone measures each tray of drill core by hand to record where every run starts and ends, its depth, and where it cracks.

Computer vision

From a tray photo to one strip

  • Gemini proposes one box per core run, as normalised 0–1000 coordinates that are clamped and mapped back to pixels; truncated JSON replies are salvaged instead of failing the run.
  • SAM 2 turns each box into a pixel mask.
  • Each mask is used as the alpha channel of a tight crop, and the crops are laid left to right in shelf order and centred on one transparent strip, so tray background never reaches the output.
  • A depth interval is spread across the strip in proportion to each segment's pixel width, with labelled boundaries.
  • A second Gemini pass finds cracks; their boxes are projected into strip coordinates and drawn on the depth strip.

Model architecture

Why SAM 2 with box prompts

SAM 2 encodes the image once with its Hiera image encoder; each box then goes through the light prompt encoder and mask decoder. One expensive encoding serves every core run in the tray, and the box prompt pins each mask to one run. The pipeline uses the small Hiera-S checkpoint (SAM 2.1).

Results

  • 20+ pixel-accurate masks per image across 100+ core samples.

Code

Python
An installable package: CLI pipeline, SAM 2 wrapper, strip stitching, depth layout and rendering, a Streamlit app, and tests for box normalisation.

Code available on request

WATonomous ASD project · Sep 2025

WATO ASD Robot

Point-to-point navigation for a simulated differential-drive robot: LiDAR costmap, A* planner and pure-pursuit control in a Dockerised ROS 2 and Gazebo stack.

  • Robotics
WATO costmap, top view: the 30 by 30 metre Gazebo world rasterised to 0.5 metre cells with a 1.5 metre inflation radius, and an A-star path from the robot's start around the central cylinder to the goal.
Reconstructed from robot_env.sdf and robot.urdf · 1.5 m inflation.

Robotics

A full navigation stack, one ROS 2 node per job

  • Costmap: each LiDAR scan becomes a 300 × 300 occupancy grid at 0.1 m per cell; every hit is inflated over 1.5 m with cost falling off linearly with distance.
  • Map memory: local costmaps are rotated by the robot's yaw and fused into a world-fixed map, only after the robot has moved far enough to add new information.
  • Planner: A* with a Euclidean heuristic; a start or goal that lands inside an obstacle is moved to the nearest free cell within three cells.
  • Control: pure pursuit at 10 Hz with a 1 m lookahead and curvature κ = 2y / L², angular speed clamped to 1.5 rad/s and a 0.2 m goal tolerance.

Results

  • 100% collision-free across the simulated test scenarios.

Code

C++
The costmap, map-memory, A* planner and pure-pursuit controller nodes (rclcpp).

Repository

ESP32-S3 + ROS 2 build · Sep 2026 – Oct 2026

Trash Can Robot

An ESP32-S3 robot that carries a trash can: the camera streams to a ROS 2 stack for inference and control, and the firmware drives two 12 V gearmotors.

  • Robotics
  • Firmware & electronics
Trash Can Robot drawn from its OpenSCAD parameters: top view of the 357 mm chassis with two drive wheels, a rear caster, battery bank, ESP32-S3 camera and the can footprint; side view with the 400 mm can.
Every dimension from cad/params.scad.

Robotics

ROS 2 on the laptop

  • camera_node decodes the MJPEG stream with OpenCV and publishes raw and compressed image topics for inference.
  • motor_node turns velocity commands into UDP motor packets, re-sent at 20 Hz.
  • safety_monitor clamps speeds and publishes zero if commands go stale (0.5 s) or the ESP32's heartbeat is lost (3 s).
  • Foxglove for live visualisation.

Firmware & electronics

ESP32-S3 firmware that fails safe

  • One FreeRTOS task owns the motor pins. Commands arrive over UDP, are guarded by a spinlock, and expire after 300 ms, so a dropped link or a dead laptop stops the robot instead of leaving it driving.
  • 5 kHz, 8-bit LEDC PWM drives an L298N H-bridge for two 12 V gearmotors (differential drive).
  • The OV3660 camera streams VGA MJPEG over HTTP from two frame buffers in PSRAM. Health is served on a separate port, because the stream blocks the first server's worker.
  • A 10 s task watchdog, plus reboots after 60 s without WiFi or 10 straight camera failures; a 1 Hz UDP heartbeat; mDNS; its own access point if the network won't join.
  • Pins avoid the S3's strapping, USB, UART0 and octal-PSRAM pins.

Firmware & electronics

Power

12 V from a USB-C PD trigger feeds the motor driver; an LM2596 buck, set to 5.0 V before the ESP32 is connected, powers the board, all on one shared ground.

Code

C++
ESP32-S3 firmware (Arduino on PlatformIO): motor task, camera, HTTP servers, UDP, watchdog, serial test commands.
Python
The ROS 2 Humble nodes and drive scripts.
OpenSCAD
The parametric chassis; every dimension lives in one parameters file.

Repository

Hack-Nation Challenge 05 · Oct 2026

The Flight of the Buffalo

An evidence engine for how rare diseases connect, with a lab-automation step that is checked in MuJoCo before anything runs.

  • Robotics
  • Systems
TFOTB workcell from its MuJoCo scene, top view: the passing pipette path visits tiprack, source and plate; the blocked path runs into the reagent tower and the simulator reports contact.
Replayed from scene.xml and the trajectory fixtures · simulation only.

Problem

Researchers looking at a rare disease want to know which other diseases share its symptoms, genes and findings, and AI-extracted claims are only useful if they can be traced to the paper.

Robotics

The lab step is simulated before it runs

  • A MuJoCo gantry workcell: three slide joints (x, y, z) with set ranges carry a pipette that collides with a reagent tower between source and plate.
  • A deterministic compiler turns an experiment spec into typed waypoints with a height for each operation, plus a ledger of liquid volumes and tips.
  • Six checks run before simulation: spec schema, operation order, pipette capacity, source volume, tip availability and declared bounds. It rejects bad specs rather than clipping them.
  • Fixtures cover a valid transfer, a blocked path, too little volume, no tips and an out-of-range move; the blocked run shown fails on purpose.

Systems

The model proposes, code decides

  • A model reads the open-access full text and proposes typed claims with strict structured output.
  • A claim is kept only if its quote appears word for word in the paper, names both entities, isn't negated, and both names resolve to MONDO, HGNC, HPO, GO or ChEBI IDs. Everything else is quarantined with its reason.
  • A cross-paper hypothesis must cite stored claims from at least two papers, and is always labelled a hypothesis.

Results

  • 52 source-quoted claims with DOIs.
  • 2.48 s median for the five-call journey on the live app (10 runs).

Code

Python
The evidence engine, paper ingestion and checks, API, and the robotics compiler and simulator.
TypeScript
The explorer front end: graph, evidence drawer and simulation view.

Repository

Own project · Mar 2026 – Apr 2026

Distance Scorer

Turns pixels-per-millimetre metadata into a calibrated apparent-distance score for soil photos, then trains a PyTorch regressor to predict it from the image alone.

  • Computer vision
  • Model training
  • Model architecture
Distance Scorer calibration from its real summary statistics: 3,405 images span 8.8 to 136.2 pixels per millimetre; the score is the z-scored natural log of PPM, so 34.5 PPM maps to 0.
From summary_stats.csv and score_config_applied.json.

Computer vision

Building the dataset

Three soil and aggregate image sources (CloseRange, GeoM, SandSnap) are reformatted into one dataset; SAM segments the soil region of each image, and zoomed crops are generated with their pixels-per-millimetre adjusted.

Model training

Training without leaking

  • Target: the natural log of pixels per millimetre, z-scored, so 34.5 PPM maps to 0 across 3,405 images.
  • 70/15/15 split grouped by parent image, so a photo and its zoomed crops never land on both sides.
  • Each sample's loss is weighted by label confidence (1.0 / 0.8 / 0.5).
  • Brightness, contrast, blur and noise augmentation; flips stay off unless they suit the domain. Weight decay 0.01, LR 1e-3 decaying to 1e-6, early stopping after 10 epochs without improvement.

Model architecture

A regressor that knows when it's unsure

  • A pretrained timm backbone (EfficientNet-B0; ConvNeXt-Tiny or ResNet-34 swap in) → pooling → MLP (256 hidden, dropout 0.2).
  • Evidential head: four outputs parameterise a Normal–Inverse-Gamma (γ, ν, α, β), with softplus keeping ν, β > 0 and α > 1. γ is the prediction.
  • Loss: the NIG negative log-likelihood (a Student-t predictive) plus a regulariser that penalises high evidence when |y − γ| is large, so confident wrong answers cost more.
  • Evaluation reports error against uncertainty: MAE by uncertainty decile and a risk–coverage table for keeping only the most certain predictions.

Results

  • Calibrated on 3,405 images spanning 8.8–136.2 PPM. Prediction accuracy not reported yet.

Code

Python
PyTorch training and evaluation, the evidential loss, dataset building and SAM segmentation, scoring calibration, a synthetic benchmark, and pytest in CI.

Code available on request

Building from the Farm, SF Tech Week 2026 · Oct 2026

Baseline

Live deepfake-voice and scam-call detection that listens to both sides of a phone call.

  • Model training
  • Model architecture
  • Systems
Baseline during a call: an amber warning, the live transcript and per-detector signals.
Screenshot of the running app during the scripted demo call.
21 s screen recording, scripted call, no sound.

Problem

Cloned-voice scam calls sound like family, and people send money before they doubt the caller.

Model training

Fine-tuning a speech + text model on an RTX 5090

  • Stage A fine-tunes the top 4 layers of WavLM-base-plus and the top 2 of DeBERTa-v3-base, with acoustic scalars and modality dropout, on emotion, arousal and delivery.
  • 55,150 segments from 2,251 calls across AMI, MELD, RAVDESS, LibriSpeech, Diplomacy and synthetic ElevenLabs calls, each dataset weighted equally per epoch.
  • Separate learning rates for encoders and heads (2.8e-5 / 1.4e-3), 5% warm-up then cosine decay, bf16 autocast, fused AdamW.
  • On held-out groups, fusing audio and text beat either alone: emotion 0.623 accuracy (text only 0.547, audio only 0.482, majority 0.437); arousal 0.798 (majority 0.587).
  • Found why one head collapsed: a class weight on a class with no training labels, combined with label smoothing, pushed it to always predict that class.

Model training

A deepfake classifier that can't cheat on identity

  • Frozen WavLM embeddings, mean-pooled over 3 s windows (1.5 s hop), into a standardised logistic regression.
  • A whole speaker is held out for testing, so the model has to learn real vs fake rather than who is speaking.
  • Telephone augmentation: an 8 kHz round trip band-limits training audio like a phone line. Reports AUC and equal error rate per window and per file.

Model architecture

Signals first, readings second

Per-turn signals (pitch, loudness, pace, pauses, fillers) and their change against the speaker's own earlier turns feed Stage B, a causal GRU over the conversation. It only sees the past, so it can run live, and it adds optional readings that report insufficient evidence when they have too few labels.

Systems

The live pipeline

Silero voice activity → local Whisper transcript, ECAPA voice match, deepfake score and delivery against the speaker's own baseline, plus LLM and rule-based fraud patterns, fused into one trust score with live warnings and a signed report.

Results

  • Warns at 0:21 in the scripted demo call and holds the transfer.

Code

Python
Detectors, fusion, the training notebooks and scripts, FastAPI back end and evaluation.
TypeScript
The React call UI and report pages.

Repository

Own project · Oct 2026

GymBuddy

An AI-native iOS gym and nutrition coach: a deterministic, unit-tested Swift engine plus Claude meal parsing from text, voice and photos.

  • Model training
  • Systems
GymBuddy profile screen: tap a muscle group on the pixel-art frog Hops to log an exercise (simulator, sample data).
Simulator screenshot, sample data.

Model training

Learned food pairings

A food2vec model trained offline over recipes scores which ingredients go together; the on-phone generator fills protein, carb, vegetable and flavour slots by pairing score minus a nutrition penalty, then a linear program solves the grams within each ingredient's portion bounds.

Systems

A deterministic engine under the AI

The coaching maths (progression, e1RM trends, trend weight, TDEE, macro targets, program planning) is a pure Swift package with no UI or AI code, so it is unit-tested on its own. Claude parses meals from text, voice and photos and writes meal plans, but the back end sums each day's macros itself and asks for one revision if a day is more than 5% off on calories or 10 g off on protein.

Results

  • 73 GymBuddyCore unit tests pass (swift test) and 15 end-to-end UI flow tests pass in the simulator (Oct 2026).

Code

Swift
SwiftUI app, the coaching engine package, widgets, a rest-timer Live Activity and Apple Health sync.
TypeScript
Cloudflare Worker: food search, barcode lookup and the Claude endpoints.
Python
Offline training of the food-pairing model.

Code available on request