Multilingual Speech Transcription Platform

Project overview

We built a multilingual speech transcription platform for a media production company that records interviews, podcasts and panel shows. Producers upload audio or video, and the platform returns a transcript that identifies who spoke when, with word-level timestamps and export formats for subtitles, show notes and archives.

The engagement covered the audio processing pipeline, speech recognition and diarization, a browser-based transcript editor and integrations with the client's editing workflow, delivered through our AI development services team. Editors work in the browser, while heavy processing stays on the company's own GPU servers.

The client and the challenge

Transcripts were produced by freelancers or generic online tools. Turnaround was slow before deadlines, costs grew with every episode, and the generic tools mixed up speakers, struggled when guests switched between languages mid-sentence and could not be trusted with unreleased content.

The client wanted fast first drafts in the languages they publish in, correct speaker labels for multi-guest recordings, precise timing for subtitles and a simple way for editors to fix names and terminology. Recordings also had to stay on infrastructure the company controlled, because many were embargoed until broadcast. Audio quality ranged from studio microphones to phone interviews and noisy location recordings, and some episodes ran long with several guests talking over one another.

Solution architecture & delivery

Uploads go into a job queue. The pipeline first normalises audio, separates it from video, applies noise reduction and uses voice activity detection to split the recording into speech segments, which avoids wasting recognition time on music and silence.

Recognition runs on Whisper models hosted on the client's own GPU servers. Language is detected per segment rather than per file, so bilingual conversations are transcribed in the language actually spoken. Alongside recognition, a PyTorch speaker-embedding model clusters voices across the recording, and the diarization output is aligned with Whisper's word timestamps so each word is attributed to a speaker. Overlapping speech is marked rather than forced onto one speaker, so editors can see where two people talked at once.

A custom vocabulary list per show, with presenter names, brands and recurring terms, is used to prompt recognition and to post-correct common misspellings. Segments where the model is unsure are highlighted in the editor so reviewers look there first instead of re-listening to the whole file. Corrections editors make to names and terms are suggested as additions to the show vocabulary, so the same mistake is less likely to recur in the next episode.

The web editor plays audio in sync with the text, lets editors rename speakers once for the entire file and exports subtitles, plain text and time-coded documents. Processing jobs are retried automatically if a worker fails, and deployments follow our CI/CD and testing pipeline. Access is restricted per show, and source files can be deleted automatically once a transcript is approved, in line with the company's retention policy.

Key features we built

• Upload of audio and video files with a processing queue and job status
• Per-segment language detection for multilingual recordings
• Speaker diarization aligned to word-level timestamps
• Show-specific vocabulary for names and terminology
• Low-confidence highlighting to focus review time
• Synchronised audio and text editor with speaker renaming
• Subtitle, plain text and time-coded exports
• Self-hosted processing so unreleased recordings stay private

How we delivered it

1. Discovery. Review of typical recordings, languages, output formats and the editorial workflow from recording to publication. We also agreed how embargoed material must be stored and who may access it.
2. Model evaluation. Whisper model sizes and diarization approaches compared on a sample set of the client's own recordings, balancing quality against GPU cost.
3. Build. Two-week sprints delivering the processing pipeline, API and editor, with exports shaped around the tools editors already used.
4. Pilot. One podcast team used the platform for new episodes, feeding back vocabulary and diarization issues before wider release.
5. Roll-out and support. Onboarding of further shows, GPU capacity tuning and model upgrades under a support agreement.

Technology choices

• Python — the natural home for audio processing, model inference and the job workers, with one codebase for the whole pipeline.
• Whisper — robust multilingual speech recognition with word timestamps that could be self-hosted, keeping embargoed audio in-house.
• PyTorch — runs the speaker-embedding and clustering models for diarization and shares GPU workers with recognition.

Outcomes

• Editors start from a speaker-labelled draft instead of typing transcripts from scratch
• Bilingual conversations are transcribed in the language actually spoken
• Subtitles and show notes are produced from the same corrected transcript
• Review effort focuses on highlighted low-confidence passages
• Unreleased recordings never leave infrastructure the company controls, and approved source files are cleared on schedule
• Transcription is available on demand before deadlines rather than waiting on freelancers

FAQ