Whisper Transcription Runbook
Whisper Transcription Runbook
Turn audio and video into clean, timestamped manuscripts on your own hardware. This walks the pipeline built on WhisperX (faster-whisper backend): extract the audio, transcribe, align word timestamps, label speakers, and batch a whole folder — no API, no upload.
Choose your stack
Pick a tool by your hardware and whether you need speaker labels, then a model size by the accuracy/speed trade-off.
| Tool | Pick when |
|---|---|
| WhisperX | You want word-level timestamps and speaker labels. Built on faster-whisper. This runbook. |
| faster-whisper | NVIDIA GPU + Python, plain transcript. ~4x faster, ~40% less VRAM. The default engine. |
| whisper.cpp | Apple Silicon, embedded, or no-Python CPU boxes. Metal-accelerated on Mac. |
| openai-whisper | The reference implementation; simplest, slowest. |
| insanely-fast-whisper | Huge batch jobs with FlashAttention-2 on a big GPU. |
| Model | ~VRAM | Use |
|---|---|---|
| tiny / base | ~1 GB | Quick drafts, real-time on CPU, low accuracy. |
| small | ~2 GB | Reasonable CPU option. |
| medium | ~5 GB | Good balance for modest GPUs. |
| large-v3 | ~10 GB | Best accuracy; multilingual and low-resource languages. |
| large-v3-turbo | ~6 GB | Distilled large-v3, ~6–8x faster. English-first; small multilingual trade-off. |
Rule of thumb Use large-v3-turbo for English and mixed everyday audio; switch to large-v3 when the material is multilingual or a low-resource language, where turbo’s trimmed decoder costs accuracy.
Install
ffmpeg does the audio/video decoding; everything else lives in a virtualenv so it does not collide with system Python.
# ffmpeg (Debian/Ubuntu; use dnf/brew elsewhere) sudo apt-get install -y ffmpeg python3-venv # Isolated environment python3 -m venv ~/whisper && source ~/whisper/bin/activate pip install --upgrade pip # WhisperX pulls in torch + faster-whisper pip install whisperx
GPU users WhisperX runs on faster-whisper, which needs matching CUDA + cuDNN. If you hit a cuDNN load error, install the cuDNN runtime that matches your CUDA/torch build. No GPU is fine — add –device cpu –compute_type int8 to every command.
Prepare the media
Whisper works at 16 kHz mono internally. The tools accept mp4/mp3/mkv directly (ffmpeg runs under the hood), but pre-extracting is faster for batch and avoids surprises with odd codecs.
# From a video ffmpeg -i talk.mp4 -vn -ar 16000 -ac 1 -c:a pcm_s16le talk.wav # From any audio file ffmpeg -i episode.m4a -ar 16000 -ac 1 -c:a pcm_s16le episode.wav
Optional but worth it -af loudnorm evens out quiet/loud passages, and -af highpass=f=100,lowpass=f=8000 trims rumble and hiss — both reduce hallucinated text on noisy source audio.
First transcription
# GPU, English, plain manuscript whisperx talk.wav --model large-v3-turbo --language en \ --device cuda --output_dir ./out --output_format txt # You can point straight at a video too whisperx lecture.mp4 --model large-v3-turbo --language en \ --device cuda --output_dir ./out --output_format txt # CPU / low-VRAM fallback whisperx talk.wav --model medium --device cpu \ --compute_type int8 --batch_size 4 --output_dir ./out
Force the language Passing –language en skips auto-detection: faster, and it avoids the model guessing wrong on a short or noisy opening.
Manuscript formats
WhisperX aligns word-level timestamps (via wav2vec2), so the subtitle and JSON outputs are tightly synced — roughly ten times tighter than vanilla Whisper.
# One format... whisperx interview.mp4 --model large-v3-turbo --language en \ --output_format srt --output_dir ./out # ...or all of them at once (txt srt vtt json tsv) whisperx interview.mp4 --model large-v3-turbo --language en \ --output_format all --output_dir ./out
Which to keep txt for the readable manuscript, srt/vtt for video captions, json when you will post-process (word timings, confidence, speakers) in a script.
Speaker labels (diarization)
Diarization answers “who spoke when”, turning a flat transcript into an interview-style manuscript with speaker turns.
Diarization uses pyannote, which is gated. Accept its license once and create a read token.
| Step | Where |
|---|---|
| Accept the model license | hf.co/pyannote/speaker-diarization-3.1 |
| Accept the segmentation license | hf.co/pyannote/segmentation-3.0 |
| Create a read token | hf.co/settings/tokens |
whisperx interview.wav --model large-v3-turbo --language en \ --diarize --hf_token hf_XXXXXXXX \ --min_speakers 2 --max_speakers 4 \ --output_format srt --output_dir ./out
Help it out Giving –min_speakers / –max_speakers when you know the count sharply improves labelling. Lines come out tagged SPEAKER_00, SPEAKER_01, which you can rename in a pass afterward.
Batch a whole folder
Point it at a directory of recordings — a local folder or a mounted NAS share — and let it grind through, skipping anything already done so you can stop and resume.
#!/usr/bin/env bash
IN=/mnt/nas/media/audio # can be a mounted Synology share
OUT=/mnt/nas/media/transcripts
mkdir -p "$OUT"
shopt -s nullglob nocaseglob
for f in "$IN"/*.{mp3,m4a,wav,mp4,mkv,mov}; do
base=$(basename "${f%.*}")
[ -f "$OUT/$base.txt" ] && { echo "skip $base"; continue; }
echo "== $base"
whisperx "$f" --model large-v3-turbo --language en \
--device cuda --output_dir "$OUT" --output_format txt
doneLong queues Load the model once per run, not per file — the loop above already does that by keeping one process. For very large libraries, transcribe first, then run a separate diarization pass (see Optimize) to avoid VRAM spikes.
Automate
Turn the batch script into a service: whenever a file lands in the inbox, transcribe it. inotifywait is the simplest trigger.
sudo apt-get install -y inotify-tools
inotifywait -m -e close_write --format '%w%f' /mnt/nas/media/inbox |
while read -r f; do
whisperx "$f" --model large-v3-turbo --language en \
--device cuda --output_dir /mnt/nas/media/transcripts \
--output_format txt srt
doneMake it durable Wrap the loop in a systemd service (or a scheduled task) so it restarts on boot and after crashes. On a Synology, a scheduled task pointing at the batch script covers most needs without a watcher.
Common flags
| Flag | Does |
|---|---|
| –model NAME | tiny | base | small | medium | large-v3 | large-v3-turbo. |
| –language en | Force language; skips detection, faster and safer. |
| –device cuda|cpu | Run on GPU or CPU. |
| –compute_type | float16 (GPU) | int8 (CPU / low-VRAM) | float32. |
| –batch_size N | Throughput vs VRAM; lower it if you hit OOM. |
| –output_format | txt | srt | vtt | json | tsv | all. |
| –output_dir DIR | Where results are written. |
| –diarize | Add speaker labels (needs –hf_token). |
| –min_speakers / –max_speakers | Bound the speaker count for better labels. |
Output formats
| Format | Use |
|---|---|
| txt | Plain readable manuscript. |
| srt / vtt | Subtitles for video players and the web. |
| json | Word timings, confidence, speaker IDs — for scripting. |
| tsv | start / end / text columns; easy to load in a sheet. |
Troubleshooting
| Symptom | Fix |
|---|---|
| Repeated / hallucinated phrases | Usually silence or music. WhisperX’s VAD filters most; also trim dead air and clean audio (Phase 2). |
| CUDA out of memory | Lower –batch_size (16→8→4), use –compute_type int8, and diarize in a separate pass. |
| cuDNN / CUDA load error | Install the cuDNN runtime matching your CUDA + torch build. |
| Wrong language detected | Always pass –language for short or code-switching clips. |
| Drifting timestamps | WhisperX alignment fixes this; vanilla Whisper drifts on long files. |
| Very slow on CPU | Use int8, a smaller model, or switch to whisper.cpp; always feed 16 kHz mono. |
| Diarization does nothing | Token missing or license not accepted — recheck the two pyannote pages. |
Optimize
| Area | Lever |
|---|---|
| Language | Pass –language when known — skips detection, avoids misdetects. |
| Model | large-v3-turbo for English speed; large-v3 for multilingual / low-resource accuracy. |
| Input | Extract 16 kHz mono up front; normalize and filter noisy audio. |
| Precision | int8 quantization: big VRAM/CPU win, minimal accuracy cost. |
| Silence | Keep VAD on — it removes dead air, cutting time and hallucinations. |
| Diarization | Run it as a second pass so pyannote and the ASR model do not spike VRAM together. |
| Throughput | One long-lived process per batch; raise –batch_size until VRAM says no. |
| Hardware | A modest NVIDIA GPU transforms throughput; on Apple Silicon use whisper.cpp + Metal. |
References
| Resource | Use | Link |
|---|---|---|
| m-bain/whisperX | Alignment + diarization pipeline | github.com/m-bain/whisperX |
| faster-whisper | The CTranslate2 backend | SYSTRAN/faster-whisper |
| openai/whisper | Reference model + model card | github.com/openai/whisper |
| whisper.cpp | C++ / Metal / CPU build | ggml-org/whisper.cpp |
