Consent-Based Voice Cloning for Dubbing

Project overview

We built a voice cloning platform for a studio that produces podcasts and dubs video content. Hosts and voice artists record a short reference session, and producers can then generate new lines in that voice for pickups, corrections and translated versions, without calling the talent back into the booth for every change.

The engagement covered the consent and enrolment workflow, voice model pipeline, a script-to-audio studio tool, search across generated takes and safeguards against misuse. It was delivered by our AI voice development team and pairs naturally with our speech transcription platform.

The client and the challenge

Small script changes meant re-booking talent and studio time. A corrected sponsor read or a mispronounced name could hold up an episode, and dubbed versions required separate voice actors who sounded nothing like the original host.

The studio wanted natural-sounding generated speech in its hosts' voices across languages, but was equally clear about the risks. Voices could only be cloned with documented permission, talent needed control over how their voice was used, and every generated file had to be traceable if it ever appeared out of context. Generated audio also had to blend with the host's real recordings closely enough that listeners would not notice a change in tone between sentences.

Solution architecture & delivery

Enrolment starts with consent. The speaker signs a digital agreement defining permitted uses, languages and duration, then records a scripted reference session that includes a spoken consent statement. The system checks that the consent recording and reference audio come from the same voice before a voice profile is created, and talent can revoke permission at any time, which disables generation immediately. Reference recordings are stored encrypted and separately from the generated library.

Voice generation uses Tortoise TTS conditioned on the speaker's cleaned reference clips. Because Tortoise favours quality over speed, generation runs as queued jobs on GPU workers: producers submit a script, the text is normalised and split into sentences, several candidate takes are generated per line and the producer picks the most natural one in the studio tool. Pronunciation overrides handle names and brand terms. Reference clips are selected per speaking style, such as conversational or advert-read, so a sponsor segment does not sound like a casual interview.

For translated versions, producers supply the approved translated script and the same voice profile is used to render it, with timing guides from the original audio so lines fit the video. Every generated file carries an inaudible watermark and a record of who requested it, the script, the voice profile and the consent scope it fell under. Talent receive a usage report showing which scripts were generated in their voice and where they were published.

ElasticSearch indexes scripts, takes, episodes and audit records, so producers can find any previous take by line text and compliance staff can search every use of a given voice. Access follows role-based permissions, and our approach to data handling is outlined on our security and compliance page.

Key features we built

• Digital consent agreements with permitted uses, languages and expiry
• Spoken consent check matched against the enrolment recording
• Instant revocation that disables a voice profile
• Script-to-speech generation with multiple candidate takes per line
• Pronunciation overrides for names and brand terms
• Translated script rendering in the original speaker's voice
• Inaudible watermarking and full generation audit trail
• Searchable library of scripts, takes and voice usage

How we delivered it

1. Discovery. Sessions with producers, talent representatives and the studio's legal adviser to define consent terms, permitted uses and misuse scenarios.
2. Voice quality trials. Reference recording guidelines and generation settings tested with volunteer hosts, comparing takes in blind listening sessions.
3. Build. Two-week sprints delivering enrolment and consent, the generation queue, studio tool and search, with safeguards built before any production use.
4. Pilot. Generated pickups used on selected podcast episodes, with hosts approving every line before release. Listener complaints and host feedback were tracked to decide which content types suited generated audio.
5. Roll-out and support. Dubbing workflows added and GPU capacity tuned, with ongoing model and safeguard updates under a support agreement.

Technology choices

• Python — runs audio cleaning, the generation workers, watermarking and the platform API in one ecosystem.
• Tortoise TTS — high-quality, expressive voice cloning from short reference clips, well suited to offline studio rendering where quality matters more than speed.
• ElasticSearch — full-text search across scripts and takes plus fast querying of audit records for every voice profile.

Outcomes

• Script corrections and pickups no longer require re-booking talent and studio time
• Dubbed versions keep the original host's voice instead of a different actor
• Voices are only generated within documented, revocable consent
• Any generated file can be traced back to its request, script and consent scope
• Producers find earlier takes by searching line text rather than browsing folders
• Talent retains approval over how their voice is used

FAQ