Voice Agents · A Complete Voice App

Three voice modes, one clean build

A real voice app: live transcription, live translation, and a tool-calling voice agent — traced with Langfuse

three modes + a tool-calling agent + Langfuse tracing

WebRTC openai-agents WebSearchTool langfuse

Built by mui-group· Advanced high-school students

The big idea

One voice app, three things it can do

Transcribe — live speech turned into text on screen.

Translate — speak one language, hear another out loud.

Assist — a voice agent that calls tools (clock, web search) to answer.

A React frontend in app/ handles the microphone and the UI. A small FastAPI backend holds the OpenAI key and runs the web-search agent. Everything the agent does is recorded to Langfuse so you can watch it think. We build all of it here, from scratch.

GOAL Three working modes, a backend that keeps your key safe, and a Langfuse dashboard showing every agent run and translation session.

The whole system in one picture

UI, one FastAPI backend, OpenAI

Notice the one dotted arrow: the backend exports OpenTelemetry spans to Langfuse. Pull it out (no keys) and everything else still works.

The playing field

Three modes, one app

Mode	Does	Transport & model	Key trait
Transcribe	Speech → text, no reply	Browser WebRTC · `gpt-realtime-whisper`	Text only, no voice back
Translate	Speak one lang, hear another	Backend WS proxy · `gpt-realtime-translate`	Needs a backend relay
Assist	Talk to a tool-using agent	Browser WebRTC · `gpt-realtime-2.1`	Calls tools mid-conversation

One live session at a time. Switching modes remounts the panel, so the old mic always stops.

Concept · one mode at a time

The switch is one piece of state

type Mode = "transcribe" | "translate" | "assist";
const [mode, setMode] = useState<Mode>("assist");

{mode === "transcribe" && <TranscribePanel key="transcribe" />}
{mode === "translate"  && <TranslatePanel  key="translate"  />}
{mode === "assist"     && <AssistPanel     key="assist"     />}

String union = typos are compile errors.

WHY THE key PROP The key makes React unmount the old panel and mount the new one on switch. Unmount runs cleanup, which stops the mic and closes the session. You can never leave two mics fighting.

Concept · keep the key secret

One FastAPI backend, for every mode

load_dotenv(find_dotenv())          # shared .env
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")

@app.post("/token")                 # for Transcribe + Assist
async def mint_token():
    headers = {"Authorization": f"Bearer {OPENAI_API_KEY}"}
    payload = {"session": {"type": "realtime",
        "model": "gpt-realtime-2.1",
        "audio": {"output": {"voice": "marin"}}}}
    async with httpx.AsyncClient() as c:
        r = await c.post(CLIENT_SECRETS_URL,
                         headers=headers, json=payload)
    return {"value": r.json()["value"]}   # the ek_ token

One small FastAPI server serves every mode: a /token route plus a WS /translate route. The key is loaded from a .env file via python-dotenv.

Mode 1 · listen and write

Transcribe: WebRTC, text only

dc.send(JSON.stringify({
  type: "session.update",
  session: {
    type: "transcription",         // no voice back
    audio: { input: {
      format: { type: "audio/pcm", rate: 24000 },
      turn_detection: { type: "server_vad" },
      transcription: { model: "gpt-realtime-whisper" },
    } },
  },
}));

EVENT ...input_audio_transcription.completed

DO NOT CONFUSE This is the user's transcript. The assistant's words are response.output_audio_transcript.* — but in transcription mode there is no assistant, so those never fire. Audio format nests under session.audio.input at GA.

Mode 2 · an honest constraint

Translate goes through the backend proxy

1Translation is a WebSocket endpoint.

2It needs an Authorization header — browsers cannot set one on a WebSocket.

3Our ek_ token is scoped to a realtime session, not translation.

[browser] --WS /translate--> [FastAPI]
          [FastAPI] --auth WS + real key--> [OpenAI]

THE FIX The browser opens a plain socket to our backend (no header needed on that leg). The backend holds the key, opens the authenticated OpenAI socket, and relays audio + transcripts both ways.

Mode 2 · the wire

Two sessions, three outputs

conversation.item.input_audio_transcription.delta
                                 # source sidecar (you)
session.output_transcript.delta  # target translation
session.output_audio.delta       # translated audio

The same mic PCM16 feeds translation plus a gpt-realtime-whisper sidecar. Source language is auto-detected.

LIVE API GOTCHA The translation docs list a source-transcript event, but the live socket did not emit it in verification. The transcription sidecar keeps “You said” reliable. Translated audio bytes arrive in event["delta"], not event["audio"].

Mode 3 · the headline

Assist: give the agent tools

export const getTimeTool = tool({
  name: "get_time",
  description: "Get the current time, optionally " +
    "for an IANA zone. Use when asked the time.",
  parameters: z.object({
    timeZone: z.string().nullable()   // strict!
      .describe("e.g. 'Asia/Tokyo'. Null = local."),
  }),
  execute: async ({ timeZone }) => {
    const zone = timeZone ||
      Intl.DateTimeFormat().resolvedOptions().timeZone;
    const now = new Date().toLocaleTimeString("en-US",
      { timeZone: zone, hour: "2-digit", minute: "2-digit" });
    return `The time in ${zone} is ${now}.`; // observation
  },
});

description tells the model when to call it.

execute is your code; its return is the observation.

web_search calls POST /web-search; the backend runs an Agents SDK agent (a few slides on) without exposing the key.

STRICT SCHEMA Use .nullable(), not .optional(). Realtime tool params must be strict or the call is rejected.

Mode 3 · wire it up

Attach the tools, open the session

const agent = new RealtimeAgent({
  name: "Capstone Assistant",
  instructions: "Concise voice assistant. When asked " +
    "the time, call get_time and say the result. " +
    "Use web_search for current or changing facts.",
  tools: [getTimeTool, webSearchTool],
});

const session = new RealtimeSession(agent, {
  model: "gpt-realtime-2.1",
  config: {
    outputModalities: ["audio"],
    reasoning: { effort: "low" }, // snappy voice
  },
});
await session.connect({ apiKey: ephemeralKey });

Adding function tools to a voice agent is identical to a text agent.

CAUTION The voice is chosen once (in the token route) and cannot change mid-session. effort:"low" keeps latency down — see the course's API reference.

The one new concept

The ReAct loop: reason → act → observe → respond

The model cannot know the time on its own. The tool is its senses.

Make it visible

Watch the loop in the app

session.on("history_updated", (history) => {
  const { lines, tools } = flattenHistory(history);
  setTranscript(lines);   // the conversation
  setToolEvents(tools);   // ACT / OBSERVE log
});

// ONE function_call item holds both:
//   name + arguments -> "ACT"
//   output (once set) -> "OBSERVE"

The AssistPanel renders an ACT / OBSERVE log beside the transcript.

CAUTION history_updated fires on every partial word. Keep the handler cheap: transform and setState, nothing heavy.

The web-search engine

/web-search on the OpenAI Agents SDK

from agents import (Agent, Runner,
    WebSearchTool, set_default_openai_key)

# hand the SDK the real key once, at startup
set_default_openai_key(OPENAI_API_KEY,
    use_for_tracing=False)               # traces go to Langfuse

agent = Agent(
    name="Web Search Delegate",
    instructions="Always use web_search; answer in short "
        "plain text a voice assistant can read aloud.",
    model="gpt-5.6",
    tools=[WebSearchTool()],             # HOSTED web search
)

result = await Runner.run(agent, query)  # ReAct loop
answer = result.final_output             # -> {"answer": ...}

Agent owns its tools; Runner runs the reason→act→observe→respond loop for you.

WebSearchTool() is hosted — OpenAI runs the real lookup. No search API to wire, no HTML to parse.

CLEAN INTERFACE The route returns a plain {"answer": "..."}, so the browser tool stays simple: send a query, read the answer.

Observability

Trace it with Langfuse (best practice)

# startup, AFTER load_dotenv():
OpenAIAgentsInstrumentor().instrument()  # auto spans
langfuse = get_client()                  # OTel exporter

# per request: ONE trace, explicit in/out, session, tags
with telemetry.trace("assist-web-search",
        input=query, session_id=sid,
        tags=["assist","web-search"], as_type="agent") as span:
    result = await Runner.run(agent, query)
    span.set_output(result.final_output)
telemetry.flush()                        # short request

The instrumentor captures model, tokens, tool calls automatically — they nest inside your trace.

Low-cardinality name, explicit input/output (just the query + answer), session_id, tags, environment, flush.

GOTCHA WE HIT Do not set_tracing_disabled(True) — it flattens traces (drops model/tool spans). We audited a real trace: with the flag = 1 span; without = 6.

Before you deploy

Guard the paid routes

1Rate limit (always on)
A per-caller-IP cap on /token, /web-search, and /translate stops a runaway loop from draining your quota.

2Origin check on the socket
The /translate WebSocket only accepts localhost or an allow-list Origin. A browser cannot forge Origin, so this blocks cross-site abuse.

3Optional shared token
Set CAPSTONE_API_TOKEN and every paid route needs it: Authorization: Bearer for HTTP, a {"token":...} field in the /translate first message.

Why: CORS only restrains browser JavaScript. It does nothing against a plain curl script hitting your key-spending routes.

Degrades gracefully: with nothing set it runs open on localhost (just the rate limit + Origin check). The frontend sends the token when NEXT_PUBLIC_CAPSTONE_API_TOKEN is set.

CHECK GET /health reports has_api_key, telemetry, and auth (true when a shared token is required).

Ship it

Run and deploy

# terminal 1: the backend (Python)
cd .../09_capstone_openai/backend
uv sync                       # openai-agents + langfuse
uv run uvicorn src.main:app --port 8000

# terminal 2: the UI (Node)
cd ../app && cp .env.local.example .env.local
npm install && npm run dev   # localhost:3000

Check: /health shows has_api_key:true, telemetry:true, and auth.

Keys: OPENAI_API_KEY (required) + optional LANGFUSE_* in the shared .env, backend-only.

No Langfuse keys? App still runs; tracing is just off (telemetry:false).

TWO LIMITS Mic needs HTTPS (or localhost). Sessions end at ~60 min — reconnect (see the course's API reference).

What you learned

Recap: three modes, full observability

Clean interface
A few small routes with plain JSON shapes — the frontend just sends a query and reads the answer.

Agents SDK
Agent + Runner + hosted WebSearchTool run the whole search loop for you; read final_output.

Langfuse
Framework integration, good names, explicit in/out, sessions, tags, nesting, flush — degrades gracefully.

REMEMBER Import Langfuse after load_dotenv(). Do not set_tracing_disabled(True) (flat traces). Keep names low-cardinality. Set explicit input/output, not a dump of args. Flush in short requests. Run → fetch the trace → audit → fix.

The end of the course

Where to go next

1Score the answers
Add a thumbs-up/down score to each trace; filter by quality in Langfuse.

2More tools + handoffs
Give the search agent a weather/wiki tool; route between multiple agents.

3Guardrails
Add input/output checks; they appear as their own observations.

4user_id + sessions
Attribute cost and quality per user; watch full conversations.

5Deploy for real
HTTPS host, backend-only env vars, reconnect at ~55 min.

You built a real voice app across three modes and made it observable — a core production skill.

Run the app, search the web, and watch the agent think in Langfuse. 🎙️ · 09_capstone_openai/app