AdofLabs

In our B3 control, median time from the end of input to browser playback was 1637.8 ms. Streaming speech synthesis accounted for a median 611.8 ms to first audio, while the final client-audio-to-playback path measured only 4.7 ms. Across the three observed runs we saw 0/3 playback underruns. The control was still serial and turn-oriented. The numbers therefore did not tell us that realtime interaction was solved; they told us where it wasn't being lost.

We built the control because component benchmarks were becoming increasingly unhelpful. A speech recognizer can report low server latency, a language model can stream quickly and a speech synthesizer can produce first audio in hundreds of milliseconds, yet the complete interaction can still feel slow. The missing measurement is the path the human actually experiences.

Figure 1 — Instrumentation points across the live path

mic
 │
 T0 capture start
 │
audio chunks
 │
 T1 input end
 │
transport
 │
 T2 server receive
 │
speech understanding
 │
 T3 usable transcript
 │
response generation
 │
 T4 synthesis request
 │
TTS stream
 │
 T5 first audio
 │
browser
 │
 T6 playable buffer
 │
speaker
 │
 T7 playback

Measurement boundary definition.

We started at the browser because otherwise server measurements could hide capture and transport costs. The client was recording mono WebM/Opus at a 48 kHz browser sample rate and emitting audio in small chunks. Early runs let us establish the basic capture rate, payload size and round-trip behavior before speech understanding was added. That made later regressions attributable to the speech stack rather than an unknown frontend path.

Adding transcription immediately exposed another useful distinction: provider latency was not the same thing as usable transcript latency. In one early whole-file transcription run the server completed quickly but the resulting transcript was empty. From a latency dashboard that request looked successful. From the perspective of the conversational system it contributed zero useful information. That changed the metric we cared about from “request completed” to “usable state became available.”

Figure 2 — A1 → A2 → B0 → B1 → B2 → B3

A1   capture + transport
A2   + speech understanding
B0   + response path
B1   timing instrumentation
B2   streaming playback
B3   frozen serial control

Each stage added one source of latency while keeping enough of the previous path fixed to preserve comparability.

Figure 3 — B3 latency waterfall

Input end ─────────────────────────────────── Playback
             1637.8 ms median

                       TTS first audio
                       611.8 ms median

                                             client → playback
                                             4.7 ms

                       TTS completion
                       895.2 ms

                       underruns
                       0 / 3 observed

The first useful conclusion was negative: browser playback was not where we should spend engineering effort. A 4.7 ms client-audio-to-playback path inside a roughly 1.64 s end-to-end control leaves almost nothing to recover there. That sounds obvious after measurement; it was not something we wanted to assume beforehand.

The second conclusion was that synthesis needs at least two measurements. Total synthesis completion describes throughput. Time to first audio describes conversational responsiveness. A synthesizer can take longer to complete the entire utterance while still producing a good interaction if generation remains ahead of playback. Conversely, impressive total throughput does not help much if the first audible packet arrives late.

The third conclusion was architectural. B3 still waited too often. Even after streaming was introduced, the interaction was fundamentally a chain of dependencies. The system accumulated delay not because every component was individually slow, but because useful work began too late. That moved our next question away from provider-by-provider optimization and toward concurrency: what can begin before the previous stage is “finished”?

Figure 4 — Same components, different scheduling

B3 SERIAL

capture → STT → reason → tool → TTS

EXPERIMENTAL OVERLAP

capture ────────────────
       STT ─────────────
           state ─────────────
              reason ───────
                 tool ─────────────
                   speech ───────────

The control is intentionally weak evidence for full-duplex behavior. It does not demonstrate simultaneous speaking and listening, interruption recovery, tool use while speech continues, or long-lived shared state. It is simply a measured reference point against which later architectures can be compared. That distinction matters because without a frozen control, improvements in perceived speed are difficult to attribute.

We are now more interested in the amount of avoidable waiting in the critical path than in a single latency number. If transcription, state update, response planning, tool execution and synthesis can overlap without making the system incoherent, then the useful optimization target changes from component speed to scheduling and state coordination.