Live captions

Service: Speech GPU · Host: wss://gpu-live.ollalink.com/v1/translate/stream · Auth: X-NH-GPU-Key

Stream microphone or meeting audio in and receive live captions in the speaker’s language: partial text while the speaker talks and a final caption for each phrase.

Finals: a final arrives about 1 s after the speaker pauses (about 0.7 s of silence in the audio ends a phrase); about 2 % of phrases wait several seconds. Stream the microphone continuously, silence included; stopping the audio for 3 s also ends a phrase, and speech without a pause is cut at 25 s.
Languages: source_lang is optional — auto-detect among en, hi, es, fr. Pin it for de, pt, ru, ar, zh, ja, ko, it, nl and vi (auto-detect never chooses them). Turkish (tr) is not offered. Kannada, Tamil, Telugu and Bengali (kn, ta, te, bn) are refused with error unsupported_recognition_source and close code 1008.
Numbers: English, Hindi, German, Spanish, French, Portuguese and Russian finals carry spoken numbers as digits (“four forty three” → 443), also when the language is detected automatically. Set source_lang for the best accuracy — automatic detection can drop words. In other languages numbers may remain words.
Capacity: live captions and realtime speech-to-text share capacity. When it is full, a new session receives {"type":"error","code":"busy",…} and close code 1013 at once — retry with back-off. Sessions have no time limit. Details: Speech routing.

Protocol

Connect the WebSocket with the X-NH-GPU-Key header.
Send one JSON config message: {"source_lang":"en","sample_rate":16000} (source_lang optional; vocabulary, a list of up to 50 terms, is accepted and does not change the captions).
Stream 16 kHz mono s16le PCM binary frames (~0.5 s each) as the speaker talks, paced at real time.
Receive JSON events:
- {"type":"ready","source_lang":"en","translation":false,…} — session open
- {"type":"partial","text":…} — interim caption (updates live)
- {"type":"final","text":…,"lang":"en","segment_id":n} — a finalized phrase, in order
- {"type":"error","code":…,"detail":…} — busy (close 1013), unsupported_recognition_source (close 1008), unauthorized / not_approved / forbidden (key problems, close 1008); other errors carry detail only.

Minimal Python client

import asyncio, json, wave, websockets

async def run():
    w = wave.open("speech_16k_mono.wav", "rb"); pcm = w.readframes(w.getnframes())
    async with websockets.connect("wss://gpu-live.ollalink.com/v1/translate/stream",
                                  additional_headers={"X-NH-GPU-Key": "sk_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}) as ws:
        await ws.send(json.dumps({"source_lang": "en", "sample_rate": 16000}))
        async def rx():
            async for m in ws:
                e = json.loads(m); print(e.get("type"), e.get("text", ""))
        t = asyncio.create_task(rx())
        step = 16000  # 0.5 s of s16le mono
        for i in range(0, len(pcm), step):
            await ws.send(pcm[i:i+step]); await asyncio.sleep(0.5)   # realtime pace
        await asyncio.sleep(6); t.cancel()

asyncio.run(run())

Notes

Input must be 16 kHz mono PCM16. Send the config as the first message, then only binary PCM frames. Pace frames at about real time — don’t send the whole file at once.
pip install "websockets>=14". On older versions use extra_headers= instead of additional_headers=.
Convert any source audio first: ffmpeg -i input.mp3 -ar 16000 -ac 1 speech_16k_mono.wav.
One connection per speaker can be shared with every listener of that room.
Spoken output is a separate request: send the caption or its translation to Text-to-speech (English, Spanish, French, German, Brazilian Portuguese, Hindi). Spoken speech-to-speech (Realtime speech) is paused.

For recordings see Speech-to-text.