The first useful design decision in SynTalk was to stop treating a hand as one input format. A webcam gives me 21 hand landmarks, or 63 coordinate values; the glove classifier shown here consumes four flex-sensor readings. Both inputs can contribute to the same spoken phrase, but they do not describe the hand in the same mathematical space.
That distinction shaped the classifier, the training flow, and the way I use Hindsight: as a memory layer for context that should survive a session, not as a replacement for sensor processing or gesture recognition.
Two ways to say the same thing
SynTalk turns a trained set of hand gestures into text and speech. A person can use a camera, a wearable glove built around flex sensors, or both. The React client owns the live interaction: MediaPipe processes camera frames, a WebSocket carries glove frames from the Node server, and the dashboard displays recognized phrases, speaks them, and stores conversation history in Firebase. A browser-based training studio records gesture examples and saves them per user.
The server has a deliberately narrow job. It reads serial data from the glove, parses it, and broadcasts sensor frames and connection state. The browser normalizes glove readings and runs the classifier. Keeping that work in the client avoids sending every sensor frame through a server-side model and keeps the live interaction path straightforward.
The two input sources have very different shapes. A camera sample contains 21 three-dimensional landmarks. The feature extractor below makes each point relative to the wrist, producing 63 values while reducing the effect of where the hand appears in the camera frame.
export function landmarksToFeatures(landmarks) {
if (!landmarks || landmarks.length < 21) return null
const wrist = landmarks[0]
const out = []
for (const lm of landmarks) {
out.push(lm.x - wrist.x)
out.push(lm.y - wrist.y)
out.push(lm.z - wrist.z)
}
return out
}
The glove path is different. The feature extractor selects four flex-sensor values from the incoming frame:
export function sensorToFeatures(fingerData) {
if (!fingerData || fingerData.length < 5) return null
return [fingerData[1], fingerData[2], fingerData[3], fingerData[4]]
}
The frame layout matters here: this helper ignores index 0 and uses positions 1–4, so the serial parser must keep that contract consistent. That is worth documenting and testing; otherwise a changed frame format could silently drop or misorder a reading. It is also an important boundary to state precisely: although the wearable design includes an MPU6050, this feature function passes four flex readings to the hardware classifier. The IMU data is not part of this particular feature vector. Camera landmarks retain spatial shape; the four flex values describe finger bend. A classifier trained on one representation should not silently receive examples from the other.
The through-line: keep context attached to its source
I made that separation a property of the training data, not just a comment in the classifier. The client maintains a camera classifier and a hardware classifier. When it loads a user’s saved examples, it selects the classifier based on the stored source. When it saves a gesture, the document ID includes that source, so a camera gesture called water does not overwrite the glove gesture with the same label.
function getClassifier(source) {
return source === 'hardware' ? hardwareClassifier : cameraClassifier
}
The Firestore document ID follows the same source-aware rule:
const id = ${source}_${name.trim().toLowerCase().replace(/\s+/g, '_')}
const ref = doc(db, 'users', user.uid, 'gestures', id)
This small decision has consequences at every boundary: recording, persistence, preload, prediction, and troubleshooting. When a user says the glove recognizes water incorrectly, I need to know whether the examples came from that glove, not merely that a gesture with that label exists somewhere in the account. The classifier selector also treats every source other than the literal hardware as camera, so the caller must validate the source before reaching this helper. Unknown source values should not silently fall through to a valid classifier.
The ID expression above is readable, but it assumes gesture names are well-behaved. In a production implementation, I would validate the source against an allowlist and normalize or encode user-provided labels so characters such as / cannot become a Firestore path separator. I would also test that two differently formatted labels cannot unintentionally collapse to the same document ID.
The in-browser KNN classifier compares a new feature vector with saved examples using cosine similarity. It votes across the five closest examples and returns no result when the best match falls below the configured threshold. That gives the application an explicit “I don’t know” path. For assistive communication, refusing an uncertain prediction is better than confidently speaking the wrong phrase.
For the glove, the runtime also has a rule-based fallback when the trained hardware classifier does not produce a prediction. Frames must remain stable across several samples before a gesture is emitted, and a cooldown limits repeated output. In hybrid mode, a shared handler accepts camera and glove predictions, attaches the source, updates the visible session log, speaks the recognized text, and forwards the event to the rest of the dashboard.
There is a useful boundary to keep in mind when describing the system: this pipeline recognizes a trained gesture vocabulary. It should not be presented as a complete translator for every sign language, with all of its grammar, movement, facial expression, and regional variation. The distinction does not make the system less useful; it makes the claim match what the recognition pipeline actually does.
Where Hindsight fits
A classifier’s training examples answer a narrow question: “What did this gesture look like?” They do not answer “What happened the last time this person used the glove?” or “Which wording did this person choose after the system offered suggestions?” Those details can help across sessions, but they do not belong in a sensor feature vector.
I use Hindsight as the durable memory layer around those interactions. The application can retain compact, user-scoped interaction notes: the recognized gesture, whether it came from the camera or glove, whether the person corrected it, which suggestion they selected, and relevant setup context such as a calibration change. It does not need to retain a continuous stream of raw landmarks or sensor readings to remember that a user repeatedly corrected a particular phrase.
The distinction between Hindsight’s operations matters here:
Retain stores useful interaction information so it can contribute to future context.
Recall retrieves memories relevant to the current phrase or assistance request.
Reflect goes a step further: it can synthesize a response grounded in several relevant memories instead of returning a list of raw facts.
I keep recognition and memory on separate paths. The local classifier decides whether the current sensor vector matches a trained gesture. Hindsight becomes useful after recognition, when the system has a phrase, a correction, or a user choice worth remembering. When the suggestion layer needs context, it can recall relevant prior choices; when the user asks for a more considered answer based on a pattern across interactions, reflection is the more appropriate operation.
This keeps the high-frequency recognition path small. It also avoids turning every request into a prompt containing the user’s entire conversation history. I retain meaningful events, retrieve only what is relevant, and keep the memory scoped to the person whose interactions produced it. Raw sensor streams remain the classifier’s concern; durable interaction context belongs in memory.
The suggestion flow already has a useful boundary: the client waits until the partial phrase stops changing, then sends a request for continuations. The server asks the language model for five short suggestions and returns an empty list if the request fails. A Hindsight-backed version can enrich that request with relevant remembered context — for example, phrasing the user has previously selected — without sending sensor frames or the full conversation history on every keystroke.
Hindsight’s memory documentation describes the retain, recall, and reflect operations behind this approach. The Hindsight GitHub repository provides the implementation reference, and Vectorize’s overview of agent memory explains why persistent context is useful in systems that interact with the same person over time.
What an interaction looks like
Suppose a user has trained the gesture water on both the glove and the camera. With the glove, four flex readings arrive over serial. The Node server parses the frame and broadcasts it. The browser selects the hardware classifier, checks the trained examples, and uses the rule-based fallback only when the trained model has no confident match. After the gesture stabilizes, the dashboard displays and speaks the recognized label.
The camera can produce the same label through a different path. MediaPipe supplies the 21 landmarks, the camera feature extractor converts them into a wrist-relative vector, and the camera classifier makes its prediction. Because the event retains its source, the dashboard and later troubleshooting can distinguish a camera recognition from a glove recognition even when both produce the same word.
Now suppose the user continues building a phrase and chooses one of the language model’s suggested completions. That selection is more useful as a memory than dozens of raw sensor frames: it records which wording helped in that context. On a later interaction, the assistant can retrieve that relevant precedent when generating suggestions. The recognizer still handles the current pose; memory helps the assistance layer maintain continuity.
A hybrid session can accept either input source while suppressing duplicate speech for a short interval. It does not need to merge camera and glove training examples to provide one coherent communication experience. Keeping the source attached to each event gives the system a way to explain where a prediction came from.
What I learned
Similar outputs do not make inputs interchangeable. Two sensors can lead to the same phrase while requiring different feature spaces, calibration, and training examples.
Put provenance in persisted records. Source-specific identifiers and metadata make collisions less likely and debugging more concrete.
Keep the fast path small. Normalize, classify, stabilize, and emit locally. Use memory for interaction context, not for every sensor frame.
An abstention is a real result. A similarity threshold gives the system room to say “no confident match” instead of inventing certainty.
Store choices, not noise. A selected completion or explicit correction can help future assistance; an unfiltered stream of sensor frames mostly adds volume.
The engineering challenge was not making a glove and a camera agree on one universal representation. It was letting them contribute to the same communication experience without erasing what makes each input distinct. Source-aware classifiers keep the physical recognition path honest. Hindsight gives the assistance layer a way to carry useful context from one interaction to the next.
Giithub Link : https://github.com/keerthanacm57/Syntalk
Website URL : https://syn-talk.vercel.app/
Hindsight GitHub: https://github.com/vectorize-io/hindsight
Hindsight docs: https://hindsight.vectorize.io/
Vectorize agent memory: https://vectorize.io/what-is-agent-memory
Top comments (0)