Making an AI tutor feel like a conversation

01

Replies that start at the pause

Waiting for speech-to-text to confirm the end of a turn leaves all of the model's start-up time for after the learner has stopped talking.

A small energy-based detector watches the mic stream. After 340 ms of quiet, the session starts generating a reply from the transcript so far and only buffers it. If the final transcript matches, the turn adopts that reply. If the learner carries on, the call is aborted and nothing was ever sent.

• The detector uses 20 ms frames, an adaptive noise floor and hysteresis, so a soft voice in a noisy room does not read as a pause.
• A wrong guess costs one discarded model call, never a cut-off learner.
• At most three attempts per turn keep the cost bounded.

matches(utterance: string, historyLength: number): boolean {
  return (
    !this.cancelled &&
    !this.failed &&
    historyLength === this.historyLength &&
    normalizeUtterance(utterance) === this.normalized
  );
}

02

Letting the learner interrupt

People cut in when they talk. A naive interrupt, though, would fire on a cough or on the AI's own voice coming back through the phone speaker.

Interruptions use a candidate and confirm model. Voice activity opens a candidate, and real words in the transcript within 700 ms confirm it. The server then aborts the model, closes the speech stream and tells the app to drop the audio it has queued.

• Signals in the first 400 ms of AI speech are ignored, because they are far more likely to be noise or echo.
• If no words arrive in time, the candidate expires silently and the AI keeps talking.
• Exam mode is left out on purpose: a real examiner does not get interrupted.

03

Surviving a provider failure

When a speech vendor refuses a turn halfway through, because of a quota or concurrency limit, everything already sent to it is gone.

Each turn keeps its own text, so the session can open a second vendor and replay the turn there. The learner hears the reply a moment late instead of not at all.

• One fallback per turn, so two refusing vendors can never loop.
• The fallback voice only speaks English, so Vietnamese sentences stay in the transcript and are not read aloud as noise.
• Grading calls retry with jittered backoff, then switch provider behind a circuit breaker.
• A speech-to-text socket that drops mid-session reconnects with backoff.