LiveKit
The livekit-plugins-munsit package brings both halves of an Arabic voice agent to LiveKit Agents: munsit.STT for speech recognition — optimized for Arabic, with Arabic/English code-switching through the munsit-en-ar model — and munsit.TTS for natural Arabic speech, streamed so the agent starts talking on the first audio chunk.
Install & authenticate
You need Python 3.10+, a Munsit account, and a LiveKit Cloud account or self-hosted LiveKit server. Generate a key at Munsit — API Keys, then install the plugin.
Both munsit.STT and munsit.TTS read MUNSIT_API_KEY automatically. Put it in a .env.local alongside your LiveKit and LLM credentials:
.env.local (never committed to Git), use environment variables in production, and rotate keys periodically.
Quick start
Create arabic_agent.py. Munsit handles both ends of the loop, so the caller is transcribed and answered in the same Arabic register — no second vendor in the pipeline. The Silero thresholds below are tuned for real-world microphones, where post-echo-cancellation audio from the agent's own speaker can otherwise be misinterpreted as user speech.
Run it in dev mode to start a local LiveKit server, launch the agent, and get a test URL to open in your browser — allow microphone access, speak Arabic, and the agent answers in a Munsit voice. Use start for deployment against your LiveKit Cloud or self-hosted server.
console mode while developing — it runs locally with your microphone and prints transcripts to stdout, no LiveKit server required. Fastest debugging loop there is.
Speech-to-text
munsit.STT() uses streaming mode by default: it keeps a live WebSocket to Munsit, emits interim transcripts while the caller speaks, and finalizes each turn with word-level timestamps the moment the server's turn detection fires — the right choice for voice agents. Pass mode="batch" to instead buffer each utterance and POST it as one request.
| Mode | When to use | Endpoint | Latency | Word timestamps | Interim events |
|---|---|---|---|---|---|
streaming (default) | Voice agents and live captions: first partials in ~0.7 s (median), server-side turn detection. | WS /api/v1/listen | ~0.7 s median to first partial; final ~1.2 s after speech ends at the default endpointing | Yes, on final transcripts, plus an utterance-level confidence score | Yes, through INTERIM_TRANSCRIPT events |
batch | Transcribing recorded audio; pipelines where one request per utterance is preferred. | POST /api/v1/audio/transcribe | VAD detection + upload + server processing (~1–2 s for short utterances) | Yes, populated on SpeechData.words | No |
STT.recognize(audio_buffer) always uses the batch HTTP endpoint, even when mode="streaming" is configured.
Pick the model by input language — and pin it explicitly (model="munsit" or model="munsit-en-ar") so a future plugin default change never affects your agent.
| Model | Use case |
|---|---|
munsit-en-ar | Mixed Arabic-English speech with code-switching. This is the plugin default. |
munsit | Pure Arabic speech recognition (fastest, and the only model that supports custom vocabulary). |
Every constructor parameter on munsit.STT():
| Parameter | Default | Description |
|---|---|---|
mode | streaming | streaming for the live WebSocket or batch for HTTP transcription. |
model | munsit-en-ar | munsit-en-ar (Arabic-English code-switching) or munsit (pure Arabic). Applies to both modes — streaming routes the session to the matching live engine. |
api_key | env MUNSIT_API_KEY | Munsit API key. |
base_url | Munsit production WebSocket URL | Override the streaming WebSocket URL. |
batch_base_url | Munsit production HTTPS URL | Override the batch HTTP URL. |
auth_method | header | Authentication style: header, bearer, or query. |
sample_rate | 16000 | Input rate hint. Streaming negotiates 8000/16000 with the server and resamples anything else automatically (LiveKit tracks are typically 48000). |
num_channels | 1 | Number of audio channels. |
interim_results | True | Emits interim transcripts in streaming mode. |
endpointing_ms | 800 | Server-side silence window that ends a turn (100–5000). Retunable mid-session via ListenStream.configure_endpointing(). |
smart_turn | True | Gate end-of-turn on the server's semantic turn-completion model in addition to silence. |
hotwords | None | Custom vocabulary: rare terms or phrases (≤200 entries, ≤40 chars each). Works in both modes on munsit; ignored, with a warning, on munsit-en-ar. |
correlation_id | None | Session identifier echoed on every streaming event (≤128 chars). |
return_confidence | False | Batch only: include per-word confidence in timestamps. |
on_enrichment | None | Callback receiving raw per-turn Sentiment / Gender event dicts from streaming. |
language | None | Label attached to SpeechData.language; defaults to ar. |
http_session | None | Custom aiohttp.ClientSession. |
extra_query_params | None | Extra query params for the streaming WebSocket endpoint. |
Munsit accepts the API key in three places — all three work on both the batch HTTP endpoint and the streaming WebSocket. Pick whichever fits your deployment:
endpointing="server_diff"/"client_vad", finalize_after_silence_ms, energy_filter and vad_silence_ms are superseded by server-side turn detection on /api/v1/listen. Each is still accepted with a DeprecationWarning, and finalize_after_silence_ms maps onto endpointing_ms.Text-to-speech
Everything tunable on munsit.TTS(). Streaming is always on — the agent starts speaking as soon as the first audio chunk is ready. Pick a voice from the Voice Library: listen, then click "Copy Voice ID" next to the one you want.
| Parameter | Values | What it does |
|---|---|---|
voice_id | e.g. ar-uae-male-1 | The Arabic voice to synthesize with. Copy IDs from the Voice Library. |
model | faseeh-v1-preview | High-quality Arabic voice synthesis. |
stability | 0.0 – 1.0 | Voice consistency. 0.0–0.4 more expressive, creative, but can hallucinate · 0.5–0.7 balanced (recommended) · 0.8–1.0 very consistent, less variation. |
speed | 0.7 – 1.2 | Speech rate. 0.7–0.9 slower (clearer for complex content) · 1.0 normal (default) · 1.1–1.2 faster (quick responses). |
sample_rate | 8000 – 48000 | Output PCM rate requested from the API. Defaults to 48000 — the engine's native rate, so audio skips downsampling and needs no resampling for WebRTC. Lower it only for a narrowband transport such as telephony. |
dialect | e.g. emirati | Optional pronunciation hint for the selected voice. |
Rules of thumb for stability: chatbots 0.6–0.8, professional applications 0.8–1.0, creative content 0.3–0.5. You can also change voice settings at runtime with update_options:
Endpointing & recognize()
In streaming mode the server decides when a turn ends: it waits for endpointing_ms of silence, and with smart_turn enabled a semantic turn-completion model must also agree the speaker is finished — so a caller who pauses mid-thought isn't cut off. A turn always ends after 2× the silence window regardless. Both are tunable at construction, and the silence window can be retuned mid-session.
Use recognize directly when you have a recorded audio buffer — a voicemail or uploaded file — and don't need a live stream. It always hits the batch HTTP endpoint regardless of mode and returns a transcript with word-level timestamps.
Tracking turn metrics
Each conversation turn carries timing data on its ChatMessage. Subscribe to conversation_item_added to read transcription delay, end-of-turn delay, and downstream LLM/TTS metrics. All values are reported in seconds.
metrics_collected event is deprecated. New integrations should use conversation_item_added and read metrics from ChatMessage.metrics.
Troubleshooting
The failure modes we see most, and the fix for each.
| Symptom | Fix |
|---|---|
| "Invalid API Key" | Verify MUNSIT_API_KEY is available to the process running your agent: echo $MUNSIT_API_KEY. |
| "Payment Required" | Your Munsit account balance is low. Top up at app.munsit.com. |
| "Rate Limit Exceeded" | Too many requests. Implement rate limiting or contact Munsit support to raise your limits. |
| No final transcript | Batch mode finalizes after LiveKit signals end-of-speech — make sure your AgentSession includes VAD: silero.VAD.load(activation_threshold=0.6, min_speech_duration=0.3). |
| Microphone feedback loop | The agent transcribes its own TTS playback: residual audio leaks through the mic after echo cancellation, and munsit-en-ar is more sensitive to low-energy input than munsit. Tighten the VAD as above; if it persists, switch temporarily to model="munsit" to confirm it's model-specific, or run with headphones. |
| Need live captions | Use munsit.STT(mode="streaming", interim_results=True) when your UI needs transcript updates before the speaker finishes. |
| No audio output | Check: API key is valid, network is stable, microphone permissions granted, browser supports WebRTC, firewall allows WebSocket connections. |
| Poor audio quality | Raise stability to 0.8+, and check your network and audio codec support. |
Where next
Go under the hood, or wire Munsit into a different agent framework.
Working with an AI assistant? Every page is available as Markdown: add .md to the URL, or send an Accept: text/markdown header. For the whole documentation in one request, point it at llms-full.txt; the page index is llms.txt. Or use Copy Page, top right.