🎬 Clip Generator

An automated, hardware-accelerated, AI-driven highlight extraction and video clipping workstation for content creators, streamers, and video editors.

Clip Generator ingests full-length livestream VODs or local recordings (OBS, gameplay, podcasts), extracts multi-track audio, runs high-speed local GPU transcription with sound dynamics analysis (screaming, combat transients, laughing fits), and prompts modern large language models (Google Gemini, Anthropic Claude, OpenAI, xAI Grok, DeepSeek) to pinpoint the most engaging, hilarious, and viral moments. Extracted clips are rendered with NVENC/CUDA GPU acceleration, vertical 9:16 auto-cropping, VR deshake stabilization, and thumbnail generation.

---

🌟 Key Features

🎙️ Audio Intelligence & Local Whisper

• 100% Free Local GPU Transcription: Runs WhisperX (batched faster-whisper + wav2vec2 word alignment + pyannote speaker diarization) on your NVIDIA GPU (cuda / float16) or on the CPU (int8). Every faster-whisper model is selectable in Settings (tiny ... large-v3, large-v3-turbo, distil-*). Zero transcription API costs.
• Speaker Labels (optional, off by default): the transcript can say who is talking (SPEAKER_00, SPEAKER_01, ...) and the AI uses it to keep question + answer and joke + reaction together. Turn it on in Settings -> Speaker detection; it needs a free HuggingFace token (how to get one: section "Speaker detection: getting a HuggingFace token" below). With it on and no token the app refuses to start a job instead of silently skipping speakers.
• Deterministic Disk Caching: Transcriptions are hashed and cached on disk (data/transcripts/). Re-analyzing or re-cutting a video with different prompts takes seconds without re-transcribing. It can be switched off globally in Settings (Use transcript cache, on by default).
• Local-Baseline Loudness: Every transcript line gets [LOUDNESS: X%] measured against the surrounding background level (rolling ±30 s median), not the global maximum, so one huge scream no longer flattens the rest of the stream.
• Laughter / Scream Detection (YAMNet on GPU): Whisper does not transcribe laughter, so the audio is also run through Google's pretrained YAMNet AudioSet classifier (16 MB ONNX model, onnxruntime-gpu). Confident laughter / screaming / shouting frames become standalone [LAUGHTER 60%] / [SCREAM 45%] lines in the transcript (the percentage is the classifier's confidence, so 30-50% is already a real signal); short peaks 18+ dB over the local background add [LOUD 60%]. A 90-minute VOD is classified in ~2 s on an RTX GPU.
  • The model is downloaded once to data/models/yamnet.onnx from a pinned Hugging Face commit and verified against a SHA256 before use.
  • Inference runs in a separate process: onnxruntime-gpu is built for CUDA 13 while PyTorch ships CUDA 12, and both use identically named DLLs (cudnn64_9.dll, ...). Loading both into one Windows process can crash it, a child process cannot take the app down. If the GPU worker fails, the classifier retries on CPU (~12 s per 90 min); if it is unavailable altogether (e.g. offline on the first run) clipping continues without laughter/scream tags.
  • onnxruntime-gpu[cuda,cudnn] pulls its own CUDA 13 libraries (~1 GB of disk) and needs an NVIDIA driver with CUDA 13 support (580+).
• Combat & Transient Action Detection: Sharp percussive transients (gunshots, explosions, hits) are tagged [ACTION: COMBAT].
• Word-Level Timestamps: WhisperX aligns every word with a wav2vec2 model, which is what makes precise clip boundaries possible.
• Multi-Track OBS Downmixing: Automatically inspects multi-track containers via ffprobe and downmixes all channels (Mic, Discord, Game) via FFmpeg amix to ensure no speech is lost.

🧠 Multi-Provider LLM Highlight Detection

• Two-Pass Windowed Analysis: The transcript is cut into overlapping 12-minute windows (60 s overlap). Each window is analysed separately, so long streams are no longer squeezed into one request where the model loses the middle. Candidates from all windows are de-duplicated and then re-scored in one short second request, putting scores from different windows on a common scale. Short videos still go out in a single request.
• Speech-shaped transcript lines: Whisper cuts its lines arbitrarily (a phrase split between two lines, 10-second run-ons), so every boundary chosen from them was logically ragged. The AI now reads utterances: neighbouring segments are joined when there is no real pause between them and the first does not end a sentence.
• Boundary review (pass 3): each selected clip is re-examined on its own with 45 s of surrounding transcript. The model first splits the excerpt into scenes (a new scene whenever the subject changes) and then picks the consecutive scenes that form ONE self-contained moment: the payoff plus the setup it needs plus its reaction, never a scene about another topic. This fixes clips without their setup, clips that drag on, and two topics glued into one clip. Back-to-back nominations are no longer merged (only genuinely overlapping ones are). Turn the pass off with settings.review_boundaries = false in config.json (one extra short request per clip).
• DeepSeek reasoning mode (on by default): DeepSeek V4 can think before answering. The boundary review measurably needs it (without it the fast model sometimes picked the wrong scene or stretched a clip to 64 s), so settings.deepseek_thinking is true by default (also a checkbox in Settings > DeepSeek). The log always states the mode: DeepSeek reasoning mode: ON (slower, better decisions), 4 requests in parallel. It is slower (about 6 minutes of AI time for a 90-minute VOD instead of about 2, in parallel batches of 4) and uses more tokens. Turn it off for quick drafts. While the AI works the log shows every request as it starts (📨 Window 3/8: asking DeepSeek (reasoning) ...), how long each answer took, and a ⏳ Still waiting ... not frozen line every 10 seconds; Cancel takes effect immediately instead of after the current batch.
• Predictable Clip Count: The LLM only nominates and ranks candidates: every 12-minute window must return its N best moments (N follows the density target, with headroom) scored honestly 1-10, weak ones included, because gatekeeping inside a window made the model return too few. The number of exported clips is controlled in code by Target clips per hour (Settings, default 12, 0 = no limit) and a minimum score (min_clip_score, default 6): the best non-overlapping candidates win. A 1.5-hour VOD yields up to about 18 clips (fewer only if there are not enough good candidates) instead of whatever the model felt like.
• Finishing the thought: a pause is not the end of an idea (a hesitation in the middle of a sentence is a pause too, and the closing remark of a joke follows the punchline half a second later). The AI sees real silences as ⏸ 1.4s lines in the transcript (lines without one are continuous speech) and is told to end after the closing remark. In code, complete_thoughts then checks both ends against the transcript: a clip that stops inside a sentence is extended to its end (up to 6 s), a short reaction or the opening of the line right after a finished sentence is included, and a clip that starts in the middle of an unfinished sentence begins at that sentence (up to 5 s). A new subject after a real pause is never pulled in.
• Fast, sliding AI requests: the AI stages keep several requests in flight (a new one starts the moment another finishes, so one slow answer no longer stalls a whole batch), and the boundary review may add at most 20 s of setup before and 12 s of tail after a clip, which stops it from stretching every clip to about a minute. Clips that turn out to be the same moment after the review are merged instead of being cut into a long clip plus a fragment.
• Cuts inside pauses, not inside words: the LLM only proposes a moment; the exact start/end are then placed by the audio (src/core/cuts.py). Each boundary is moved to the nearest pause in the waveform (energy envelope, the way VAD-based clippers do it), because ASR timestamps drift by hundreds of milliseconds and a fixed pre/post-roll lands mid-word. On a real 90-minute VOD a blind 0.75 s / 1.25 s pad put ~50% of the cuts inside loud speech. Only real pauses count (>= 0.3 s first, then >= 0.2 s: shorter dips are gaps between syllables, 28% of cuts used to land in them and clipped a word); about 87% of boundaries find one, the rest stay on the transcript's utterance boundary. The exported audio gets a 40 ms fade in and a 150 ms fade out so a cut never sounds chopped. Before that, a coarse step snaps to utterance boundaries (2 s radius), guarantees setup before / reaction after the model-reported peak_time, lets a laugh play out and keeps neighbouring clips from overlapping.
• Universal Model Support: Native, direct API clients with automatic fallback:
  • Google Gemini: gemini-3.5-flash, gemini-3.1-pro, gemini-2.5-flash, gemini-2.5-pro (large context).
  • Anthropic Claude: claude-sonnet-5, claude-opus-5, claude-haiku-4-5.
  • OpenAI: gpt-5.5, gpt-5.4, gpt-4o, gpt-4o-mini.
  • xAI: grok-4.3, grok-4.6, grok-4-1-fast-*.
  • DeepSeek & OpenRouter: deepseek-flash, deepseek-v4-pro, or any custom OpenAI-compatible endpoint.
• Dynamic Model Discovery: Automatically queries provider endpoints in the background to keep the model picker updated with the latest available releases.
• Configurable Prompt Profiles: The built-in Default prompt documents the audio tags, decides clip length purely by content (no target or maximum-length anchor: clips range from a few seconds to a few minutes), has an explicit cutting section and worked examples, an absolute 1-10 rubric, and "clip it" voice-command recognition. The protected Default profile is re-synced from the codebase on every launch; custom profiles are left untouched.

⚡ GPU-Accelerated Video Pipeline

• Hardware Encoding (NVENC / AMF / CUDA): Renders output MP4 clips using dedicated hardware encoders (h264_nvenc) for blazing-fast export speeds.
• Vertical 9:16 Auto-Cropping: Ready-to-publish exports for TikTok, YouTube Shorts, and Instagram Reels:
  • Standard Center Crop
  • Left-Third (Facecam)
  • Right-Third
  • Blurred Background (Portrait)
  • Custom Coordinate Crop
• VR Deshake Filter: Optional post-processing filter to stabilize jittery head-tracking motion in VR gameplay recordings.
• Thumbnail & Metadata Generation: Automatically produces .jpg poster frames and .json sidecars containing timestamps, duration, virality score, and AI justification for each clip.

🖥️ Desktop Workstation

• Modern Dark UI: Built with CustomTkinter for a sleek, responsive interface.
• Batch Video Clipper: Select and process local video files or recordings in an efficient queue.
• Structured, colour-coded log console: the run is divided into numbered stages (══ 2/4 · 🤖 AI finds and ranks the best moments ══), details hang under their stage, and colours mean the same everywhere: cyan = transcription/audio, violet = AI, teal = selection and statistics, orange = cutting/export, green = success, amber = warning/cancel, red = error, grey = background progress. Log text is shown literally (never interpreted as HTML), and the final line reports the total run time.
• Built-in Clip Gallery: Review clips directly in the application, sort by Date or Virality Score, view AI reasoning notes, play video highlights, and batch delete unwanted cuts.
• Discord Webhook Alerts: Sends rich notifications to your Discord channels when clipping jobs complete.

---

🏗️ Architecture & Pipeline

flowchart LR
    A[Raw Local Video / Recording] --> B[FFmpeg Audio Ingestion]
    B -->|amix Multi-Track| C[16kHz PCM Buffer]
    C --> D[Local GPU WhisperX + word timestamps + speakers]
    C --> E[Local-baseline loudness + combat + YAMNet laughter/scream events]
    D & E --> F[Annotated Transcript]
    F --> G1[LLM: candidates per 12-min window]
    G1 --> G2[Dedupe + global re-score]
    G2 --> G3[Density selection + boundary refinement]
    G3 -->|Timestamps & Virality Scores| H[GPU FFmpeg Video Engine]
    H -->|NVENC + 9:16 Crop + VR Deshake| I[Rendered MP4 Clips]
    H --> J[Thumbnails & JSON Metadata]
    I & J --> K[Clip Gallery / Discord Webhook]

Loading

---

📋 System Requirements & Prerequisites

Minimum Requirements

• Operating System: Windows 10/11 (64-bit)
• Python: 3.12 or 3.13 (WhisperX 3.8 requires < 3.14)
• Package Manager: uv (strongly recommended) and make

Hardware Recommendations

• GPU: NVIDIA GeForce RTX / GTX series (driver with CUDA 13 support, 580+) for local WhisperX acceleration (float16), YAMNet sound classification and NVENC video exports.
• RAM: 16 GB+
• Disk Space: High-speed SSD storage for video scratch buffers and caches.

Required External Binaries

The application expects the following helper executables inside the bin/ folder (or available in your system PATH):

Binary	Purpose	Download Source
ffmpeg.exe	Video cutting, encoding, audio filters	gyan.dev FFmpeg Builds
ffprobe.exe	Audio/video stream inspection	Included with FFmpeg

---

🚀 Installation & Quick Start

1. Clone the Repository

git clone https://github.com/jBahrVR/jBahrs-Clip-Generator.git
cd jBahrs-Clip-Generator

2. Place External Binaries

Ensure ffmpeg.exe and ffprobe.exe are located in the bin/ directory:

jBahrs-Clip-Generator/
├── bin/
│   ├── ffmpeg.exe
│   └── ffprobe.exe

3. Setup with make (Recommended)

This project uses make and uv for virtual environment management, PyTorch CUDA wheel resolution, and dependency isolation.

# Create virtual environment and install all runtime + dev dependencies:
make install

# Launch the application:
make run

---

🛠️ Alternative: Manual Setup (without make)

If make is not installed on your system, you can use uv directly:

# 1. Create a virtual environment
uv venv .venv

# 2. Activate virtual environment
.venv\Scripts\activate

# 3. Install dependencies with PyTorch CUDA 12.9 support
#    (unsafe-best-match lets uv take numpy & co. from PyPI while torch comes from the CUDA index)
uv pip install --index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cu129 -r requirements.txt -r requirements-dev.txt
#    faster-whisper installs the CPU `onnxruntime`, which shares its module with `onnxruntime-gpu`: put the GPU build back on top
uv pip install --reinstall --no-deps onnxruntime-gpu==1.30.0

# 4. Launch the application
python main.py

---

🧪 Development, Code Quality & Tests

Run the complete test suite, coverage reporting, linter, and dead-code analysis with a single command:

make test

This target executes:

1. Pytest Test Suite with Coverage: python -m pytest -v --cov=src --cov=main --cov-report=term-missing tests/
2. Ruff Linter & Formatter Check: python -m ruff check main.py src tests
3. Dead Code Analysis: python -m vulture main.py src tests --min-confidence 80 --ignore-names event,icon,cls

---

📖 Usage Guide

1. Initial Configuration

1. Launch the application (make run).
2. Navigate to the ⚙️ Settings tab.
3. Configure your paths:
   • Generated Clips Directory: Destination folder for exported highlight clips.
4. Add your API Key(s) under the AI Providers section:
   • Google AI Studio (Gemini): Recommended for multi-hour recordings due to the massive context window (2M tokens).
   • Anthropic / OpenAI / xAI / DeepSeek: Supported out-of-the-box.
5. Click "Save Settings" to persist changes.

2. Video Highlight Extraction

1. Go to the 🎬 Clip Extractor tab.
2. Select local video file(s) (.mp4, .mkv, .avi, .mov, .flv) using the Browse Files button or enter file paths.
3. Choose the active Prompt Profile in Prompt Manager and Whisper Model (any faster-whisper model: tiny, base, small, medium, large-v3, large-v3-turbo, distil-*, ...). In Settings, tune Target clips per hour to control how many clips you get (more clips = lower score bar; 0 disables the limit).
4. Click "Process Files" to begin analysis and export.

3. Clip Gallery

1. Go to the 🖼️ Clip Gallery tab.
2. Sort extracted clips by Date or Virality Score.
3. Select any item to view generated thumbnails, audio duration, and the AI's written justification.
4. Click ▶️ Play Clip to preview the cut in your default media player.

---

🗣️ Speaker detection: getting a HuggingFace token

Speaker detection (diarization) uses the pyannote/speaker-diarization-community-1 model. It is free, but the model is gated: you need a HuggingFace account, you must accept the model's terms once, and the app needs an access token. It is optional; everything else works without it.

1. Create a free account at https://huggingface.co/join (or log in).
2. Accept the model terms. Open https://huggingface.co/pyannote/speaker-diarization-community-1, fill in the short form if it asks and press Agree and access repository. Without this step the download fails with a 401 / 403 error even with a valid token.
3. Create a token. Open https://huggingface.co/settings/tokens, press Create new token, choose the type Read, give it any name (for example clipgen) and press Create token. Copy the token now (it starts with hf_ and is shown only once).
4. Paste it into the app. Settings -> Speaker detection: tick Detect speakers, paste the token into HuggingFace token (the field unlocks when the box is ticked) and press Save settings. The Get a token button opens the token page.

Notes:

• The first run downloads the diarization models (a few hundred MB) into data/models/huggingface.
• Instead of the field you can set the HF_TOKEN environment variable or put the token into a file data/.hf_token.
• With the box ticked and no token anywhere the app will not start processing and tells you why; untick the box to work without speakers.
• Speaker detection adds time and memory on long streams (the whole recording is processed at once). If it fails, the transcript is made without speakers and the reason is logged.
• A leaked token can be revoked on the same tokens page. The token is stored in plain text in config.json (git-ignored).

---

⚙️ Configuration Reference

Everything the app stores lives inside the project folder (nothing is kept in your user profile or on another drive location):

⚠️ config.json in the project root holds your API keys and HuggingFace token in plain text. It is git-ignored, but do not zip, share or cloud-sync it. data/ (git-ignored, with its own .gitignore) holds several GB. The app points HF_HOME / TORCH_HOME / NLTK_DATA into data/models, overriding any global setting. Speaker diarization runs on the whole recording at once: on very long streams it can run out of memory, in which case the transcript is made without speakers (logged).

• config.json (project root, next to the Makefile) — API keys, HuggingFace token, active provider, model preferences, crop dimensions, storage paths. A config.json left by an older release in %APPDATA%/jBahrsClipGenerator/ is copied here once; the old folder can then be deleted.
• data/models/ — the downloaded AI models: whisper/ (faster-whisper), huggingface/ (alignment + pyannote diarization), torch/, nltk/, yamnet.onnx (sound classifier).
• data/transcripts/ — Cached transcripts, one JSON document per video: {version, engine, model, language, diarized, speakers, segments} (segments carry speaker, word timestamps, loudness and audio events). The cache is versioned and keyed by the video, Whisper model, language and whether speakers could be labelled; transcripts of older releases are not reused. settings.use_transcript_cache (default true) turns it off globally: nothing is read or written.
• logs/ (project root, git-ignored) — Timestamped execution and error logs. data/.hf_token — optional file with the HuggingFace token.
• The AI reads the transcript as JSON Lines, one object per line (speaker, start, end, text, optional loudness, combat, pause_before, and {"event": "LAUGHTER", ...} lines). The built-in Default prompt explains this format; custom prompt profiles written for the old [start - end] text format must be updated by hand.
• settings.clips_per_hour (default 12, 0 = unlimited) and settings.min_clip_score (default 6) in config.json control how many clips are exported. settings.review_boundaries (default true) switches the scene-based boundary review, settings.deepseek_thinking (default true) DeepSeek's reasoning mode.

---

💡 Model Recommendations

Provider	Model	Best For	Notes
Google	gemini-3.5-flash	All-round Best	Large context and fast; windowed analysis keeps quality high on 4+ hour VODs.
Anthropic	claude-sonnet-5	High Precision	Exceptional nuanced comedy and banter comprehension.
DeepSeek	deepseek-flash	Budget / Value	Cost-effective extraction via official API or OpenRouter.
OpenAI	gpt-5.5	General Gaming	Ensure your account tier supports large token requests.
xAI	grok-4.3	Edgy / Fast Comedy	Strong grasp of chaotic banter and sarcasm.

---

🤝 Origin & Credits

This project originated from the prototype concept created by jBahrVR.

The current codebase has been entirely re-engineered from the ground up into a production-grade desktop application featuring modular architecture, multi-provider API integrations, native CUDA hardware acceleration, dynamic model discovery, multi-track audio downmixing, comprehensive unit testing, and automated system tray workflows.