# LiveKit

> The livekit-plugins-munsit package brings both halves of an Arabic voice agent to LiveKit Agents: munsit.STT for speech recognition — optimized for Arabic, with Arabic/English code-switching through the munsit-en-ar model — and munsit.TTS for natural Arabic speech, streamed so the agent starts talking on the first audio chunk.

## Install & authenticate

You need Python 3.10+, a [Munsit account](https://app.munsit.com/), and a LiveKit Cloud account or self-hosted LiveKit server. Generate a key at [Munsit — API Keys](https://app.munsit.com/en/api-keys), then install the plugin.

```
pip install livekit-plugins-munsit
```

```
git clone https://github.com/CNTXTFZCO0/livekit-plugins-munsit.git
cd livekit-plugins-munsit
pip install -e .
```

Both `munsit.STT` and `munsit.TTS` read `MUNSIT_API_KEY` automatically. Put it in a `.env.local` alongside your LiveKit and LLM credentials:

```
# Munsit — speech-to-text and text-to-speech
MUNSIT_API_KEY=your_MUNSIT_API_KEY_here

# LiveKit Configuration
LIVEKIT_URL=wss://your-livekit-server.com
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

# LLM Configuration (for the agent's brain)
OPENAI_API_KEY=your_openai_api_key
```

> Your API key is only shown once. Save it securely. Keep it in .env.local (never committed to Git), use environment variables in production, and rotate keys periodically.

## Quick start

Create `arabic_agent.py`. Munsit handles both ends of the loop, so the caller is transcribed and answered in the same Arabic register — no second vendor in the pipeline. The Silero thresholds below are tuned for real-world microphones, where post-echo-cancellation audio from the agent's own speaker can otherwise be misinterpreted as user speech.

```
"""
Arabic voice assistant — Munsit STT + Munsit TTS
"""
from dotenv import load_dotenv
from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession
from livekit.plugins import munsit, openai, silero

load_dotenv(".env.local")

class ArabicAssistant(Agent):
    """Arabic-speaking voice assistant"""

    def __init__(self) -> None:
        super().__init__(
            instructions="""أنت مساعد صوتي ذكي يتحدث العربية بطلاقة.
            مهمتك مساعدة المستخدمين بالإجابة على أسئلتهم بطريقة واضحة ومفيدة.
            كن ودوداً ومحترماً في تعاملك."""
        )

server = AgentServer()

@server.rtc_session()
async def my_agent(ctx: agents.JobContext):
    session = AgentSession(
        stt=munsit.STT(model="munsit-en-ar"),   # Arabic + English code-switch
        llm=openai.LLM(model="gpt-4o", temperature=0.7),
        tts=munsit.TTS(
            voice_id="ar-uae-male-1",          # Copy IDs from the Voice Library
            model="faseeh-v1-preview",
            stability=0.75,
            speed=1.0,
        ),
        vad=silero.VAD.load(activation_threshold=0.6, min_speech_duration=0.3),
    )

    await session.start(room=ctx.room, agent=ArabicAssistant())

    await session.generate_reply(
        instructions="رحب بالمستخدم باللغة العربية وقدم نفسك كمساعد ذكي جاهز للمساعدة."
    )

if __name__ == "__main__":
    agents.cli.run_app(server)
```

Run it in dev mode to start a local LiveKit server, launch the agent, and get a test URL to open in your browser — allow microphone access, speak Arabic, and the agent answers in a Munsit voice. Use `start` for deployment against your LiveKit Cloud or self-hosted server.

```
python arabic_agent.py dev
```

```
python arabic_agent.py start
```

> Tip: use console mode while developing — it runs locally with your microphone and prints transcripts to stdout, no LiveKit server required. Fastest debugging loop there is.

## Speech-to-text

`munsit.STT()` uses **streaming mode** by default: it keeps a live WebSocket to Munsit, emits interim transcripts while the caller speaks, and finalizes each turn with word-level timestamps the moment the server's turn detection fires — the right choice for voice agents. Pass `mode="batch"` to instead buffer each utterance and POST it as one request.

| Mode | When to use | Endpoint | Latency | Word timestamps | Interim events |
| --- | --- | --- | --- | --- | --- |
| `streaming` (default) | Voice agents and live captions: first partials in ~0.7 s (median), server-side turn detection. | `WS /api/v1/listen` | ~0.7 s median to first partial; final ~1.2 s after speech ends at the default endpointing | Yes, on final transcripts, plus an utterance-level confidence score | Yes, through `INTERIM_TRANSCRIPT` events |
| `batch` | Transcribing recorded audio; pipelines where one request per utterance is preferred. | `POST /api/v1/audio/transcribe` | VAD detection + upload + server processing (~1–2 s for short utterances) | Yes, populated on `SpeechData.words` | No |

```
from livekit.plugins import munsit

streaming_stt = munsit.STT()
batch_stt = munsit.STT(mode="batch")
```

> STT.recognize(audio_buffer) always uses the batch HTTP endpoint, even when mode="streaming" is configured.

Pick the model by input language — and pin it explicitly (`model="munsit"` or `model="munsit-en-ar"`) so a future plugin default change never affects your agent.

| Model | Use case |
| --- | --- |
| `munsit-en-ar` | Mixed Arabic-English speech with code-switching. **This is the plugin default.** |
| `munsit` | Pure Arabic speech recognition (fastest, and the only model that supports custom vocabulary). |

Every constructor parameter on `munsit.STT()`:

| Parameter | Default | Description |
| --- | --- | --- |
| `mode` | `streaming` | `streaming` for the live WebSocket or `batch` for HTTP transcription. |
| `model` | `munsit-en-ar` | `munsit-en-ar` (Arabic-English code-switching) or `munsit` (pure Arabic). Applies to both modes — streaming routes the session to the matching live engine. |
| `api_key` | env `MUNSIT_API_KEY` | Munsit API key. |
| `base_url` | Munsit production WebSocket URL | Override the streaming WebSocket URL. |
| `batch_base_url` | Munsit production HTTPS URL | Override the batch HTTP URL. |
| `auth_method` | `header` | Authentication style: `header`, `bearer`, or `query`. |
| `sample_rate` | `16000` | Input rate hint. Streaming negotiates 8000/16000 with the server and resamples anything else automatically (LiveKit tracks are typically 48000). |
| `num_channels` | `1` | Number of audio channels. |
| `interim_results` | `True` | Emits interim transcripts in streaming mode. |
| `endpointing_ms` | `800` | Server-side silence window that ends a turn (100–5000). Retunable mid-session via `ListenStream.configure_endpointing()`. |
| `smart_turn` | `True` | Gate end-of-turn on the server's semantic turn-completion model in addition to silence. |
| `hotwords` | `None` | Custom vocabulary: rare terms or phrases (≤200 entries, ≤40 chars each). Works in both modes on `munsit`; ignored, with a warning, on `munsit-en-ar`. |
| `correlation_id` | `None` | Session identifier echoed on every streaming event (≤128 chars). |
| `return_confidence` | `False` | Batch only: include per-word confidence in `timestamps`. |
| `on_enrichment` | `None` | Callback receiving raw per-turn `Sentiment` / `Gender` event dicts from streaming. |
| `language` | `None` | Label attached to `SpeechData.language`; defaults to `ar`. |
| `http_session` | `None` | Custom `aiohttp.ClientSession`. |
| `extra_query_params` | `None` | Extra query params for the streaming WebSocket endpoint. |

Munsit accepts the API key in three places — all three work on both the batch HTTP endpoint and the streaming WebSocket. Pick whichever fits your deployment:

```
# Default: sends the key as the x-api-key header.
munsit.STT(auth_method="header")

# Authorization: Bearer <key>
munsit.STT(auth_method="bearer")

# Query parameter (?token=<key>): useful when an upstream proxy strips headers.
munsit.STT(auth_method="query")
```

> The 0.3.x streaming parameters are deprecated. endpointing="server_diff"/"client_vad", finalize_after_silence_ms, energy_filter and vad_silence_ms are superseded by server-side turn detection on /api/v1/listen. Each is still accepted with a DeprecationWarning, and finalize_after_silence_ms maps onto endpointing_ms.

## Text-to-speech

Everything tunable on `munsit.TTS()`. Streaming is always on — the agent starts speaking as soon as the first audio chunk is ready. Pick a voice from the [Voice Library](/text-to-speech/voices): listen, then click **"Copy Voice ID"** next to the one you want.

| Parameter | Values | What it does |
| --- | --- | --- |
| `voice_id` | e.g. `ar-uae-male-1` | The Arabic voice to synthesize with. Copy IDs from the [Voice Library](/text-to-speech/voices). |
| `model` | `faseeh-v1-preview` | High-quality Arabic voice synthesis. |
| `stability` | `0.0` – `1.0` | Voice consistency. **0.0–0.4** more expressive, creative, but can hallucinate · **0.5–0.7** balanced (recommended) · **0.8–1.0** very consistent, less variation. |
| `speed` | `0.7` – `1.2` | Speech rate. **0.7–0.9** slower (clearer for complex content) · **1.0** normal (default) · **1.1–1.2** faster (quick responses). |
| `sample_rate` | `8000` – `48000` | Output PCM rate requested from the API. Defaults to `48000` — the engine's native rate, so audio skips downsampling and needs no resampling for WebRTC. Lower it only for a narrowband transport such as telephony. |
| `dialect` | e.g. `emirati` | Optional pronunciation hint for the selected voice. |

Rules of thumb for `stability`: chatbots **0.6–0.8**, professional applications **0.8–1.0**, creative content **0.3–0.5**. You can also change voice settings at runtime with `update_options`:

```
# Start with one voice
tts = munsit.TTS(voice_id="ar-uae-male-1", speed=1.0)

# Switch to a different voice based on context
tts.update_options(
    voice_id="ar-hijazi-female-2",
    stability=0.8,
    speed=1.0
)
```

## Endpointing & recognize()

In streaming mode the **server** decides when a turn ends: it waits for `endpointing_ms` of silence, and with `smart_turn` enabled a semantic turn-completion model must also agree the speaker is finished — so a caller who pauses mid-thought isn't cut off. A turn always ends after 2× the silence window regardless. Both are tunable at construction, and the silence window can be retuned mid-session.

```
# Snappier finals: shorter silence window, semantic gating still on.
stt = munsit.STT(
    mode="streaming",
    endpointing_ms=300,
    smart_turn=True,
)

# Retune mid-session on the live stream (100–5000 ms).
stream = stt.stream()
await stream.configure_endpointing(800)
```

```
from livekit import rtc
from livekit.plugins import munsit

stt = munsit.STT()
frames = [...]  # list of rtc.AudioFrame
combined = rtc.combine_audio_frames(frames)

result = await stt.recognize(combined)
print(result.alternatives[0].text)
for word in result.alternatives[0].words:
    print(f"{word.start_time:.2f}s -> {word.end_time:.2f}s  {word}")
```

Use `recognize` directly when you have a recorded audio buffer — a voicemail or uploaded file — and don't need a live stream. It always hits the batch HTTP endpoint regardless of `mode` and returns a transcript with word-level timestamps.

## Tracking turn metrics

Each conversation turn carries timing data on its `ChatMessage`. Subscribe to `conversation_item_added` to read transcription delay, end-of-turn delay, and downstream LLM/TTS metrics. All values are reported in seconds.

```
from livekit.agents import ChatMessage

@session.on("conversation_item_added")
def on_item(event):
    msg = event.item
    if not isinstance(msg, ChatMessage):
        return

    metrics = msg.metrics or {}
    if msg.role == "user":
        transcription_delay = metrics.get("transcription_delay")
        end_of_turn_delay = metrics.get("end_of_turn_delay")
        if transcription_delay is not None:
            print(f"STT delay: {transcription_delay * 1000:.0f} ms")
        if end_of_turn_delay is not None:
            print(f"EOU delay: {end_of_turn_delay * 1000:.0f} ms")
    elif msg.role == "assistant":
        llm_ttft = metrics.get("llm_node_ttft")
        tts_ttfb = metrics.get("tts_node_ttfb")
        if llm_ttft:
            print(f"LLM TTFT: {llm_ttft * 1000:.0f} ms")
        if tts_ttfb:
            print(f"TTS TTFB: {tts_ttfb * 1000:.0f} ms")
```

> The previous metrics_collected event is deprecated. New integrations should use conversation_item_added and read metrics from ChatMessage.metrics.

## Troubleshooting

The failure modes we see most, and the fix for each.

| Symptom | Fix |
| --- | --- |
| **"Invalid API Key"** | Verify `MUNSIT_API_KEY` is available to the process running your agent: `echo $MUNSIT_API_KEY`. |
| **"Payment Required"** | Your Munsit account balance is low. Top up at [app.munsit.com](https://app.munsit.com/). |
| **"Rate Limit Exceeded"** | Too many requests. Implement rate limiting or contact [Munsit support](/support) to raise your limits. |
| **No final transcript** | Batch mode finalizes after LiveKit signals end-of-speech — make sure your `AgentSession` includes VAD: `silero.VAD.load(activation_threshold=0.6, min_speech_duration=0.3)`. |
| **Microphone feedback loop** | The agent transcribes its own TTS playback: residual audio leaks through the mic after echo cancellation, and `munsit-en-ar` is more sensitive to low-energy input than `munsit`. Tighten the VAD as above; if it persists, switch temporarily to `model="munsit"` to confirm it's model-specific, or run with headphones. |
| **Need live captions** | Use `munsit.STT(mode="streaming", interim_results=True)` when your UI needs transcript updates before the speaker finishes. |
| **No audio output** | Check: API key is valid, network is stable, microphone permissions granted, browser supports WebRTC, firewall allows WebSocket connections. |
| **Poor audio quality** | Raise `stability` to 0.8+, and check your network and audio codec support. |

> Support: livekit-plugins-munsit on PyPI · GitHub Issues · LiveKit Community · Munsit support. Apache 2.0 licensed.

## Where next

Go under the hood, or wire Munsit into a different agent framework.

- [STT streaming API](/speech-to-text/streaming) — The WebSocket that streaming mode rides on. — `WSS /api/v1/listen`

- [Batch transcription API](/speech-to-text/transcribe) — The HTTP endpoint behind batch mode and recognize(). — `POST /api/v1/audio/transcribe`

- [Browse voices](/text-to-speech/voices) — Listen to every Arabic voice and copy its ID. — `GET /voices`

- [Pipecat plugin](/integrations/pipecat) — The same Munsit STT and TTS in a different agent framework. — `pipecat-plugins-munsit`