SignBridge

An offline, two-way ASL fingerspelling translator for Deaf patients and their doctors — built at a hackathon.

Point a webcam at a hand fingerspelling the ASL alphabet (A–Y, no J/Z — those need motion) and SignBridge reads the letters back in real time, entirely on-device. The doctor can reply by voice or text. Nothing leaves the machine: no network calls, no cloud LLM, no data collection.

How it works

• Hand tracking: MediaPipe Hands locates the hand and landmarks in the browser.
• Letter classification: a small CNN (Conv-BN-Pool ×2 → Dense → Dense), trained on the Sign Language MNIST dataset, classifies the cropped 28×28 handshape — running as a pure-JS forward pass in public/cnn.js using weights exported to public/model_cnn.json.
• UI: public/app.js wires up the camera, live letter readout, message buffer, doctor reply panel (voice + text), a practice/learning mode, and a debug panel showing model accuracy on held-out test images.
• Server: server.js is a minimal static file server — it only serves public/, no backend logic.

Test accuracy on the held-out set: 94.8%.

Running it

npm start

Then open http://localhost:5173.

Retraining the model

python train_cnn.py <train.csv> <test.csv>

Reproduces the CNN from Copy_of_ASL_Sign_Language_Workshop.ipynb and re-exports weights to public/model_cnn.json. Set FULL=1 for the slower, notebook-faithful training run with data augmentation.