Foxy Junior is an English adventure where speaking moves the story forward, so the app has to judge a young voice, survive a bad network and delete the recording as soon as it is scored.
Role
Flutter app and NestJS backend
Platforms
iOS, Android
Stack
Flutter, NestJS, Prisma, Azure Speech
Try it
The problem
A game that listens
A speaking turn is a hard thing to put in front of a young child. Small fingers drift off a button, sentences trail away before the finger lifts, and a speech service can hear nothing, time out or score a long sentence poorly. None of those should feel like failing.
The audio is also the most sensitive data the product touches. A child's recording has to reach a scoring service and then disappear from the device, even if the app is killed mid-turn, and the text the recogniser heard has to stay out of analytics, crash reports and the database.
Architecture
The phone owns the speaking turn
The Flutter client runs the whole turn. It records a 16 kHz mono WAV, posts it directly to Azure Pronunciation Assessment and decides the outcome on the device. The NestJS backend never receives audio. Its speech job is to exchange the Azure key for a 10-minute token, behind an app key, a validated device id and rate limits.
I put the pass, retry and fallback ladder in one pure function so it can be tested without a widget, a microphone or a scorer. Scored turns go into an on-device queue and reach Postgres later, as numbers and reference words only, with a client event id so a retry cannot write a duplicate.
Capture
Flutter
Hold-to-speak take, 16 kHz mono WAV
Token broker
NestJS
Exchanges the Azure key for a 10-minute token
Scoring
Azure Speech
Pronunciation scores for the sentence and each word
Decision
Dart
Passed, retry, silent or fallback, decided on device
Score log
Prisma, Postgres
Scores and reference words, no audio or transcript
The recording travels from the phone to Azure directly. The backend only sees the token request and, later, the numbers.
Key decisions
Three calls around a microphone
01
The button answers the finger
Opening a microphone takes several platform round trips, and a child can let go before it is running. Hold-to-speak then turned into tap to record, tap again to stop.
The screen flips to recording on the touch itself while the recorder catches up, and a flag remembers a release that came during opening, so the take stops the moment the mic is ready. The button reads raw pointer events, because a gesture recognizer cancels when a finger drifts. After release the mic stays open 400 ms longer, so the last syllable is still in the take.
A press under 500 ms earns a nudge to hold longer and costs no attempt.
Voice and music stop before recording starts.
The iOS audio session is warmed up on arrival at the step.
02
Only real attempts cost anything
A plain pass or fail collapses very different events: a mumble, a silent take, a timeout, a long sentence the recogniser scores low. Treating them alike would punish a child for a home network.
One pure function sorts each take into passed, retry, silent or fallback. A technical failure passes with the full reward and no attempt spent. When Azure hears nothing the child is asked again at no cost, and after 3 silent takes in a row, or the step's attempts, the child moves on with no star and half the XP.
The pass line is 60 out of 100, and a long sentence can pass on its best keyword score.
A whole scoring attempt is capped at 15 seconds, so the thinking state always ends.
Only speaking steps can trigger the fallback, so a touch game cannot cost a star.
03
A voice that cannot linger
The recording plugin writes to disk, so a child's audio exists as a file between the mic closing and the scorer answering. Azure also returns a transcript and lists extra words the child said, which can include their own name.
Each take gets a unique temp path, is deleted as soon as the scorer has read it, and is deleted again in a finally block on every way out. A sweep at startup catches what a process kill left behind. The transcript is stopped by allowlists on the device and again on the server, so a new field loses data instead of leaking it.
Extra words are dropped in the app, the DTO and the repository.
The speech log table has no audio or transcript column.
Crash reports run with screenshots and session replay off.
