Skip to content

OPERATOR-ROUTING.md

Latest commit

 

History

History
239 lines (163 loc) · 11.1 KB

OPERATOR-ROUTING.md

File metadata and controls

239 lines (163 loc) · 11.1 KB
Raw

Legends YT-DLP operator routing (router-loaded recipe)

Part of the cto-legends ecosystem. Loaded by the router on demand; this file carries no skill registration.

Operate the local Legends YT-DLP CLI as the deterministic runtime. This recipe is the operator brain; the CLI is the control plane.

Start Here

  1. Confirm the working directory is the project root containing pyproject.toml.
  2. Read wiki/hot.md if it exists.
  3. Run:
legends-yt-dlp doctor

For VPN-guarded runs, add doctor --production and pass --with-vpn to plan, preflight, and run. Mullvad is optional; ordinary doctor, preflight, and run do not inspect or require Mullvad, a subscription, or an account file. Bulk acknowledgement does not enable VPN.

If the command fails, fix setup before planning or running downloads.

Safety Boundary

Allowed: lawful archiving, Mullvad privacy checks, conservative pacing, resumable batches, optional rights metadata, and operator-reviewed retries.

Disallowed: bypassing DRM, paywalls, login challenges, captchas, access controls, account controls, throttling, bans, or IP blocks. Do not automatically rotate Mullvad relays/IPs to keep downloading through source-side blocks.

Real downloads require passing preflight plus explicit --yes. Anonymous yt-dlp operation (no browser cookies, cookie files, username/password auth, .netrc, or inherited user config) applies to every run. Mullvad posture (connected, Lockdown on, split tunneling off, LAN sharing blocked, auto-connect on) applies only with --with-vpn. Batches over 50 URLs require --bulk acknowledgement; batches of 11-50 print a pacing notice.

If the user asks for automatic IP switching on download errors, implement only this safe policy:

  • tunnel/VPN/network failure in --with-vpn mode: reconnect Mullvad, rerun preflight, retry within limits.
  • item-level transient failure: retry using yt-dlp retry settings.
  • source-side throttle, captcha, login challenge, explicit block, or repeated rate limit: pause the batch and report. Do not switch relays to continue.

For the detailed policy, read docs/SAFETY-AND-ERRORS.md.

Conversational operation

Tell the agent what you want to download. The agent handles the CLI flags, setup, and checks; you do not need to know commands such as --with-vpn. An occasional authorized pull proceeds without a VPN or a separate VPN approval step.

For a large request, such as 100 videos, the agent explains pacing and may offer VPN assistance as an optional convenience. Your explicit request for that batch supplies the bulk acknowledgement; it does not require another approval card. Declining VPN, having no subscription, or having no Mullvad installation is fine at any batch size. Bulk mode never enables VPN automatically. VPN cannot guarantee protection from throttling, blocks, or account enforcement and is not a way to bypass source-side controls. Pause and report those controls when encountered.

Core Workflows

Setup / Inspect

legends-yt-dlp onboard
legends-yt-dlp yt-dlp version
legends-yt-dlp yt-dlp update
legends-yt-dlp catalog

Use onboard first in a new checkout or after unpacking an alpha zip. It prints readiness, missing setup steps, the user interview checklist, and the next safe commands without mutating local state.

Read docs/COMMANDS.md for the full command surface.

For release verification:

python scripts/check_release.py

Inspect docs/PACKAGING.md, docs/LEGAL.md, NOTICE, and LICENSE before publishing a release package.

For VPN fail-closed testing, use only the guarded command:

legends-yt-dlp mullvad disconnect-test --emergency-unlock

Do not use raw disconnect during operator work unless the user explicitly accepts the risk. For normal end-of-work cleanup, use mullvad shutdown; it disables Lockdown before disconnecting and verifies Mullvad is no longer connected. The guarded test constrains to a US relay by default, checks Lockdown behavior, and reconnects before returning.

Post-Capture Intelligence

Use this only after a lawful local batch has been downloaded and verified. The intelligence module never downloads media, calls yt-dlp, operates Mullvad, rotates relays, or continues through source-side controls.

Do not automatically transcribe every batch. The default archive workflow stops after run, verify, and ledger review. Invoke intelligence when the user asks for transcription, search, clip extraction, transcript vaults, or when the job context clearly calls for analysis and the operator explains that next step.

legends-yt-dlp intelligence init "<manifest.json>"
legends-yt-dlp intelligence transcribe "<manifest.json>" --media "<video.mp4>" --video-id "<video-id>" --model auto
legends-yt-dlp intelligence import-crispasr "<manifest.json>" --input "<transcript.json>" --video-id "<video-id>" --media-path "<video.mp4>"
legends-yt-dlp intelligence ingest-words "<manifest.json>" --input "<words.jsonl>" --video-id "<video-id>"
legends-yt-dlp intelligence align nfa "<manifest.json>" --media "<video.mp4>" --video-id "<video-id>" --prepare-only
legends-yt-dlp intelligence search "<manifest.json>" "agentic workflow"
legends-yt-dlp intelligence clips plan "<manifest.json>" --query "agentic workflow"
legends-yt-dlp intelligence vault build "<manifest.json>"

The stable contract is a timestamped word ledger. Prefer the ready-made CrispASR Parakeet backend for local transcription. Heavy ASR stacks such as NVIDIA NeMo + Parakeet remain advanced external producers of that ledger, preferably isolated in WSL2, Docker, or a separate Python environment. Use intelligence align nfa when the user needs tighter word-boundary timing; it prepares/imports NVIDIA NeMo Forced Aligner CTM output but still keeps NeMo/PyTorch/model weights outside the core package.

Use intelligence doctor --require-gpu before describing a local ASR setup as GPU-ready. CPU-only CrispASR still runs locally and token-free, but --require-gpu must not be mixed with --no-gpu or --gpu-backend cpu.

Productized Archive SOP

Use this sequence for real channel, playlist, or larger URL-set work:

  1. Consult the user for scope, source URLs, optional rights notes/evidence, output location, and limits. Remind them to use material only when they have rights, permission, a license, fair use, or another lawful basis. Also choose folder policy: mixed ad hoc links usually belong in one batch folder; channel, playlist, or multi-source archive work usually belongs in uploader/source folders.
  2. Inventory channel or playlist URLs before downloading.
  3. Create or review the item ledger.
  4. Run preflight (add --with-vpn for guarded runs).
  5. Dry-run.
  6. Ask for explicit approval before a real run.
  7. Run with --yes.
  8. Verify artifacts.
  9. Refresh and review the item ledger.
  10. Stop on source-side throttles, captchas, login challenges, account controls, or blocks.

The safety boundary is firm: do not use relay rotation, account switching, login automation, captcha solving, or other continuation tactics to push through source controls.

Plan A Batch

For one or more explicit URLs:

legends-yt-dlp plan "<URL>" --name "<batch-name>"

For many URLs, create a text file with one URL per line and run:

legends-yt-dlp plan --from-file "<urls.txt>" --name "<batch-name>"

Attach rights evidence when available:

legends-yt-dlp plan --from-file "<urls.txt>" --rights "optional permission/license/fair-use note" --rights-file "<rights-evidence.md>" --name "<batch-name>"

For mixed hand-curated links, keep the user's working set together:

legends-yt-dlp plan --from-file "<urls.txt>" --name "<batch-name>" --folder-policy batch

Folder policies are auto, batch, by-uploader, and flat. Use batch for one folder per request. Use by-uploader for channel, playlist, or multi-source archive work.

Inventory A Channel Or Playlist

For channel or playlist work, inventory first:

legends-yt-dlp inventory "<channel-or-playlist-url>" --rights-file "<rights-evidence.md>" --name "<batch-name>"
legends-yt-dlp ledger "<manifest.json>"

Inventory creates items.jsonl for operator review without downloading media. Use --max-items, --max-height, --max-downloads, or --max-filesize when the user wants a bounded first pass.

For smoke tests or bounded jobs, add limits:

legends-yt-dlp plan "<URL>" --name "<batch-name>" --max-downloads 1 --max-height 360 --max-filesize 50M

Read docs/BATCH-CATALOG.md before planning large archives.

Validate The Install

Use a small operator-verified smoke URL before treating an install as ready:

legends-yt-dlp smoke plan --url "<authorized-video-url>" --name "first-smoke"
legends-yt-dlp preflight "<manifest.json>"
legends-yt-dlp run "<manifest.json>" --dry-run
legends-yt-dlp run "<manifest.json>" --yes
legends-yt-dlp verify "<manifest.json>"
legends-yt-dlp ledger "<manifest.json>" --refresh

Preflight / Dry Run / Real Run

legends-yt-dlp preflight "<manifest.json>"
legends-yt-dlp run "<manifest.json>" --dry-run

Only after the user explicitly approves a real download. The CLI prints a legal-use notice before invoking yt-dlp:

legends-yt-dlp run "<manifest.json>" --yes
legends-yt-dlp verify "<manifest.json>"

run --with-vpn defaults to safe VPN recovery. It may reconnect Mullvad and retry for tunnel/network failures. Use --no-recover-vpn only for diagnostics.

Dry-runs suppress raw yt-dlp JSON by default. Use --show-output only when debugging.

Concurrency Policy

Default to one active YouTube batch at a time. Do not parallelize YouTube downloads by default. Large archives should be systematic through cataloging, download archives, resume support, and pacing, not aggressive concurrency.

If the user asks about parallelism, read docs/BATCH-CATALOG.md.

Transcription Routes

Capture first; transcribe only verified local media when asked. In-module transcription runs an external CrispASR executable (never bundled). For continuous background audio or voice sketchpads, hand off to legends-ambient-intelligence; for live typing, hand off to hyperyap. Read docs/TRANSCRIPTION.md.

Empire Vault

intelligence vault build exports Obsidian-compatible transcript pages. Map them into Empire per docs/VAULT-MAP.md: cite by video_id, keep the batch manifest as rights provenance, never vault binaries.

EDR and Security Guardrails

  • Never spawn background shells, hidden console windows, or execution-policy bypass shell wrappers.
  • Never read local browser profile databases, DPAPI stores, or cookie files; all yt-dlp runs enforce --ignore-config, --no-cookies, and --no-cookies-from-browser.
  • Never automate VPN relay rotation to continue through source-side rate limits, captchas, login prompts, or blocks.

Reporting Back

Report:

  • what command was run
  • whether the run used --with-vpn (and whether Mullvad was connected)
  • batch manifest path
  • URL count
  • rights evidence path, if attached
  • item ledger path and status summary
  • whether it was dry-run or real
  • stop condition, if any
  • verify result, if a real run occurred
  • next safe action