English · 简体中文
Showcase · Pipeline · Quick start · Model servers · Web UI & CLI
DubFlow is a powerful video dubbing framework built for multi-speaker conversations and multilingual storytelling. It brings together speaker-aware voice cloning, localized lip-sync, and adaptive timing to carry a video's dialogue across languages while keeping its characters and visual identity at the heart of the experience. From solo presenters to multi-person scenes, upload a video and bring your story to a new audience.
Languages: Chinese · English · Japanese · Korean · German · French · Russian · Portuguese · Spanish · Italian
Source language is detected automatically by default; the target language is explicit.
| Original · Chinese | DubFlow · English |
|---|---|
original.mp4 |
translated_en.mp4 |
ASR + alignment recover dialogue and timing, while audiovisual speaker binding associates each line with an on-screen character. Audio separation isolates dialogue from the background; an LLM served through vLLM translates the text, TTS generates the target-language voices, and a lip-sync model served through LightX2V edits mouth motion locally. Compositing brings the new dialogue, edited regions, and background audio together. These are module roles, not fixed model choices; the setup below documents the current implementations.
Model outputs are estimates, not guarantees of verbatim translation, clean separation, or perfect synchronization. When no suitable visible face can be bound, the pipeline replaces audio without forcing a mouth edit. Execution failures are reported as failures.
Requires Linux, CUDA GPUs, FFmpeg, curl, unzip, and uv. Run commands from the repository root:
git clone https://github.com/ModelTC/DubFlow.git
cd DubFlow
# Main pipeline: recognition, separation, visual binding, and web UI
bash scripts/install_unified.sh "$PWD/.venv"
# Qwen3-TTS: isolated Transformers 4.57 environment
uv venv --python 3.11 .venv-tts
uv pip install --python .venv-tts/bin/python \
--index-url https://download.pytorch.org/whl/cu128 \
torch==2.8.0+cu128 torchaudio==2.8.0+cu128
uv pip install --python .venv-tts/bin/python \
qwen-tts==0.1.1 transformers==4.57.3 accelerate==1.12.0 soundfile
# Translation server
uv venv --python 3.11 .venv-vllm
uv pip install --python .venv-vllm/bin/python vllm==0.23.0Install a LightX2V checkout with native TBDub support in a separate environment, following that repository's instructions. Its environment also needs scikit-image for color correction.
These environments are intentionally separate because of dependency constraints. The main installation script links FFmpeg libraries; the pipeline sets the library search path when launching stages.
Accept any required model access terms and use .venv/bin/hf auth login when authentication is needed. Keep credentials out of source files.
export MODEL_DIR=/path/to/models/video_dubbing
PYTHON_BIN="$PWD/.venv/bin/python" bash scripts/download_models.sh allThis downloads the current model stack and its auxiliary weights; model files are not included in this repository. SAM-Audio Large-TV uses ModelScope by default. Set SAM_AUDIO_DOWNLOAD_SOURCE=hf to use Hugging Face.
Download individual components
# Replace "all" with one of these component groups:
# core TBDub Student, VAE, HuBERT, face detection/geometry/identity
# speech Qwen ASR, ForcedAligner, Nemotron, Qwen3-TTS
# loconet UniTalk-LoCoNet active-speaker weights
# sam-audio SAM-Audio Large-TV and its text encoder
# qwen35-translation Qwen3.5-9B
PYTHON_BIN="$PWD/.venv/bin/python" bash scripts/download_models.sh sam-audio.venv/bin/python scripts/configure.py \
--models "$MODEL_DIR" \
--python "$PWD/.venv/bin/python" \
--tts-python "$PWD/.venv-tts/bin/python" \
--output config.local.jsonThe template is config.example.json. The local file is ignored by Git; use --force only when you intend to replace an existing configuration.
Example deployment using four GPUs:
| GPU | Process | Responsibility |
|---|---|---|
| 0 | vLLM | Qwen3.5-9B translation |
| 1 | LightX2V | TBDub Student lip-sync |
| 2, 3 | DubFlow workers | One video request per worker; TTS stays in a separate resident process on the same GPU |
Start each command in a separate terminal, from the DubFlow root. Set MODEL_DIR to the same model directory in each terminal; replace the LightX2V paths with your installation.
Terminal 1 — translation
GPU_ID=0 MODEL_PATH="$MODEL_DIR/Qwen3.5-9B" \
VLLM_BIN="$PWD/.venv-vllm/bin/vllm" \
bash scripts/serve_translation.shTerminal 2 — lip-sync
GPU_ID=1 LIGHTX2V_PATH=/path/to/LightX2V \
PYTHON_BIN=/path/to/lightx2v-env/bin/python \
MODEL_PATH="$MODEL_DIR/tbdub" \
CONFIG_PATH="$PWD/configs/tbdub_server.json" \
bash scripts/serve_lipsync.shDefault endpoints are http://127.0.0.1:18440/v1 and http://127.0.0.1:18441, matching translation_url and tbdub_url in the configuration.
LightX2V and DubFlow must see the same absolute media paths, including when running in containers. Keep these internal model endpoints private. GPU memory requirements depend on the selected models and workload; the GPU IDs above are a deployment example, not a minimum-memory guarantee.
Terminal 3 — browser interface and pipeline workers
DUB_GPUS=2,3 DUB_CONFIG="$PWD/config.local.json" \
DUB_DATA_DIR="$PWD/outputs/web" bash scripts/serve.shOpen http://127.0.0.1:18320, wait for model loading, then upload a video and select the target language. Two workers can process two video jobs concurrently; the shared lip-sync service queues crop requests. Set DUB_GPUS=2 for one pipeline worker.
Optional: temporary public HTTPS access
With cloudflared installed:
CLOUDFLARED_BIN=/path/to/cloudflared PORT=18320 bash scripts/public.shUse the HTTPS URL printed in the terminal. Anyone with the link can upload and trigger inference; share it only with intended users. Stop the tunnel to remove access. Do not expose the vLLM or LightX2V endpoints.
Default uploads allow 250 MB per file and 32 MiB per chunk. Adjust DUB_MAX_UPLOAD_MB and DUB_UPLOAD_CHUNK_MB when starting the web service.
Start both model services first. Use an idle pipeline GPU, not one already assigned to a web worker:
.venv/bin/python main.py \
--config config.local.json --gpu 2 \
--source input.mp4 --target-language en \
--output outputs/example_en| Option | Behavior |
|---|---|
--target-language en |
Required output language: zh/en/ja/ko/de/fr/ru/pt/es/it |
--source-language zh |
Override automatic source-language detection |
--duration-mode adaptive_video |
Adapt eligible video and background-audio segments to speech duration; default configuration |
--duration-mode fixed |
Preserve video duration and fit translated speech into its time window |
--resume |
Resume the same request; use a new output directory after changing inputs or configuration |
final.mp4— delivered video;final_fixed.mp4is the pre-retiming intermediate in adaptive mode.source.txt/translated.txt— recognized and translated dialogue.report.json— audio replacement and lip-sync outcomes.run.json/logs/— pipeline progress and per-stage errors.
Web jobs live under $DUB_DATA_DIR/jobs/<id>/; resident model loading logs are under $DUB_DATA_DIR/resident/. Stop each foreground service with Ctrl+C.
Code layout
main.py Pipeline orchestration
stages/
prepare.py Normalize video and audio
recognition/ ASR, alignment, acoustic speaker labels
speakers/ Detection, active speaking, identity binding
separate.py Dialogue/background separation
speech/ Translation and voice synthesis
lipsync/ Crop planning and LightX2V requests
output/ Mix, paste back, retime, export
runtime/ Configuration, clients, resident processes
web/ Uploads, job queue, result viewer
scripts/ Installation, downloads, service launchers
Both the CLI and web workers use main.py. Individual stage options are available with python -m stages.<module> --help.
Built with LightX2V, vLLM, and the upstream models recorded in third_party/SOURCES.json. Original licenses and notices remain in third_party/ and licenses/. Model and input-media permissions apply separately; use voice cloning only with appropriate authorization.

