Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/sessions/260719-1155-t8-implementation/HANDOFF.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# T8 实施 session —— 交接(HANDOFF)

> 创建:2026-07-19(第二个实施 session 开头)
> 任务:T8 阶段二逐段校对推广 v1(确定性段落化),规格唯一来源:`docs/sessions/260719-0513-chapters/TASKS.md` 的 T8 节
> 分支:`feat/chapters-wiring`(本地,未 push)

## 前情:上一个 session 为什么中断

第一个实施 session 主循环累积约 60 万 token(消息体 2.1MB),超过 provider 2MB
硬上限;compaction 请求本身也超限失败,session 死锁报废(导出留档:
仓库根目录 `kimi-export-session_-20260719-161035.md`,仅供参考,不要整份喂回上下文)。
本交接文件 + git 提交历史是恢复的完整依据。

**教训(本 session 起执行)**:
- 大件探查/实现一律走 subagent,主循环只收结论,不整读几千行文件;
- 进度随时落盘到本目录;消息体接近上限前主动收工换新 session。

## 已完成(全部已提交,git log 为准)

| 步 | commit | 内容 |
|----|--------|------|
| 方案修订 | `13b00cb` | T8 方案 v1 确定性段落化,增量 review 收敛 |
| S1 | `62aca84` | 确定性段落化工具 paragraphize(长度预算+授权断点),`transcriber/paragraphize.py` |
| S2 | `812ef97` | `structured_calibration_for_plain` 开关 + 段落化/分块配置字段 |
| S5 | `0a55f09` | 渲染器无 speaker 防御修复 + plain_structured 开关回退门控 |
| S3 | `822bb1b` | SpeakerAwareProcessor has_speaker 模式(无说话人逐段校对,零 SpeakerInferencer 调用,落盘前段落化)+ 19 个单测 |

S3 提交时 `uv run pytest tests/unit` 全绿(exit 0,约 2400 项;warnings 摘要会
盖住通过数行,以 exit code 为准)。

## 待办(按序)

1. **S4 — llm_ops 接线**(`api/services/llm_ops.py`,当前未接线,grep 无
`plain_structured`/`has_speaker` 痕迹):
- `_prepare_llm_content`:**仅 `calibrate_requested=True`**(含 recalibrate)且
plain 源有 segments → 返回 list(协调器 isinstance 路由自然生效);
calibrate=false 补层任务维持纯文本(防指纹永久 mismatch → 永久 nolink);
- structured 落盘 gate(现仅 `use_speaker_recognition`)扩展到本模式;
- plain 结构化产物写 provenance `"mode": "plain_structured"`;
- `llm_calibrated.txt` 继续生成(无前缀变体,保 export/raw 兼容)。
2. **S6 — 集成测试**:开关开全链路(plain 源 → llm_processed.json 落盘 →
渲染带 `dlg-{i}` 锚点 → 章节 fingerprint 匹配 → `jump_ok=1`);
开关关回归(plain 路径行为与现状一致)。验收全集见 TASKS.md T8「完成标准」。
3. **review 循环**:独立 subagent review ≤20 轮;P1 必修,P2/P3 可判「接受不修」
记 `REVIEW-LOG.md` backlog;gate = 连续 2 轮无新增 P1。

## 纪律提醒

- 开关 `llm.structured_calibration_for_plain` 默认 **false**(暗启动,T9 验收后才翻 true);
- `llm_processed.json` 序列化全文不含 "unknown"(硬断言);
- 减法优先,新机制仅用于消除 P1;
- 勿把 worktree 本地 config 的 `min_chapters_threshold=1000` 当生产默认合入(示例默认 10000)。
17 changes: 17 additions & 0 deletions docs/sessions/260719-1155-t8-implementation/REVIEW-LOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# T8 实施 session —— review 日志与 backlog

> 创建:2026-07-19
> 任务:T8 阶段二逐段校对推广 v1(确定性段落化),任务卡 `docs/sessions/260719-0513-chapters/TASKS.md`
> 纪律:独立 subagent review ≤20 轮;P1(正确性/安全/数据丢失)必修;P2/P3 可判「接受不修」记本文件 backlog 并写理由;gate = 连续 2 轮无新增 P1;减法优先,新机制仅用于消除 P1。

## Review 轮次记录

| 轮次 | reviewer | 新增 P1 | P2/P3 新增 | 处置 |
|------|----------|---------|------------|------|
| (待第 1 轮) | | | | |

## Backlog(接受不修的 P2/P3)

| 日期 | 轮次 | 发现 | 级别 | 接受理由 |
|------|------|------|------|----------|
| (暂无) | | | | |
281 changes: 227 additions & 54 deletions src/video_transcript_api/llm/processors/speaker_aware_processor.py

Large diffs are not rendered by default.

99 changes: 99 additions & 0 deletions src/video_transcript_api/llm/prompts/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -426,6 +426,105 @@ def build_summary_user_prompt(
Return ONLY the proofread JSON, without any additional explanations or comments."""


# ============================================================
# 结构化校对任务 Prompt 模板(无说话人变体)
# ============================================================
# 与 STRUCTURED_CALIBRATE 系列同契约({id,text} 输出、禁止合并/拆分/增删/重排),
# 仅行格式描述去掉 [说话人] 括号,与 SpeakerAwareProcessor._format_chunk_for_prompt
# 的 has_speaker=False 行生成保持一致。

STRUCTURED_CALIBRATE_NO_SPEAKER_SYSTEM_PROMPT = """你是专业的中文文本校对助手。你的任务是对语音转录(ASR)文本进行逐段校对。

## 核心要求(必须遵守)

输入每一行格式为 `[id][HH:MM:SS]: 内容`(该段无时间戳时为 `[id]: 内容`),其中 `[id]` 是该段的编号(锚点)。

1. **按 id 锚点返回**:对每一段输入,返回一个 `{id, text}` 对象,`id` 与输入行的 `[id]` 完全一致
2. **每段都要返回**:即使某段无需修改,也要原样返回该 id 的 text;不要遗漏任何 id
3. **只改文本**:`text` 只包含校对后的内容,**不要**包含 id 或时间戳
4. **禁止合并、拆分、增删、重排**:一个 id 对应一段,不得把多段并入一个 text,也不得拆成多个

## 专有名词修正(最高优先级)

用户会提供一份**关键信息列表**,其中包含本次内容涉及的正确人名、品牌、术语等。
ASR 转录经常将这些专有名词错误识别为同音字或近音字。你必须:
1. **逐一检查**关键信息列表中的每个词,在转录文本中寻找可能的 ASR 误识别
2. 将 ASR 误识别的文字**替换为关键信息列表中的正确拼写**
3. 常见的 ASR 专有名词错误模式:
- 英文品牌被拆成无意义的中文字/拼音(如 "哥斯 r 哦" → "Cursor"、"欧普斯" → "Opus")
- 中文名词被识别为同音字(如 "海陆" → "海螺"、"金业酒店" → "今夜酒店")
- 英文缩写被错误拼写(如 "IR" → "ARR"、"PMFF" → "PMF")
- 英文单词被拆成多个片段(如 "mini max" → "MiniMax")

## 校对规则

1. **只能在单个段落内部进行修改**,不得跨段落操作
2. **最小改动原则**:只做必要的校对与通顺化;若有多种改法,选择更接近原句的版本
3. 修正明显的错别字和语法错误,优先修正"读起来明显不通顺"的表达
4. 调整标点符号的使用,确保其正确性和一致性
5. 如有必要,可以**适度调整语序**以提高可读性,但不要改变原意或强调重点
6. 可以**删减少量无意义的口头禅/口误/重复词**(如"那个…就是…就…"),但不要删掉关键信息或语气词
7. 可对明显口误做纠正(如把"任一/套生/压力面"等修成通顺表达)
8. 保留原文中的口语化表达和说话者的语气特点,避免过度书面化
9. **保留语义与事实**:不要改变人称、否定/肯定、因果关系、时间顺序、数字/金额/数量等核心信息
10. **不确定时偏保守**:宁可轻微修整,也不要重写或引入新含义
11. **重点修正 ASR 常见错误**(不改变原意):
- 叠字与重复:例如"遇遇不到/薪薪十多万万/约约咖啡馆"等
- 错分词或错字:例如"a 本大学/任一/身出市场/套生大学"等明显不通顺表达
- 英文字母或符号误入导致的断裂(如 a 本 → 一本)
- 同音词误识别导致的歧义(结合上下文与关键信息列表修正)
12. **保持完整性**:不要新增或删减实质内容,不要压缩或概括
13. 不要解释或评论文本内容

## 输出格式

必须输出 JSON 格式,包含 corrections 数组,每项为 `{"id": 整数, "text": "校对后文本"}`。
- `id` 与输入行的 `[id]` 一一对应,必须覆盖每一个输入 id
- `text` 只放校对后的内容,不要带 id、时间戳或行格式标签

只返回校对后的JSON,不要包含任何其他解释或评论。"""

STRUCTURED_CALIBRATE_NO_SPEAKER_SYSTEM_PROMPT_EN = """You are a professional English text proofreading assistant. Your task is to proofread speech-to-text (ASR) transcription text segment by segment.

## Core Requirements (MUST follow)

Each input line has the format `[id][HH:MM:SS]: content` (or `[id]: content` when the segment has no timestamp), where `[id]` is the anchor index of that segment.

1. **Return by id anchor**: for every input segment, return one `{id, text}` object whose `id` exactly matches the input line's `[id]`
2. **Return every segment**: even if a segment needs no change, return its text unchanged under its id; do NOT omit any id
3. **Only edit text**: `text` contains ONLY the proofread content, NOT the id or timestamp
4. **Do NOT merge, split, add, remove, or reorder segments**: one id maps to exactly one segment

## Proofreading Rules

1. **Only modify within a single segment**, do not operate across segments
2. **Minimal changes principle**: only make necessary corrections; when multiple options exist, choose the one closest to the original
3. Fix obvious typos and grammar errors, prioritizing expressions that are clearly unnatural
4. Adjust punctuation for correctness and consistency
5. If necessary, **slightly adjust word order** to improve readability, but do not change the meaning
6. May **remove minor filler words/speech errors/repetitions** (e.g., "um", "like like", "you know you know"), but do not remove key information
7. Correct obvious speech errors and ASR misrecognitions (e.g., homophones, run-on words, incorrectly split words)
8. Preserve colloquial expressions and the speaker's tone/style, avoid over-formalizing
9. **Preserve semantics and facts**: do not change person, negation/affirmation, causality, temporal order, numbers/amounts
10. **When uncertain, be conservative**: prefer minor polishing over rewriting or introducing new meaning
11. **Focus on common ASR errors** (without changing meaning):
- Homophones: e.g., "their/there/they're", "your/you're", "its/it's"
- Run-on or incorrectly split words
- Missing or incorrect punctuation from speech-to-text
- Misrecognized proper nouns (correct using reference information)
12. **Maintain completeness**: do not add or remove substantive content, do not compress or summarize
13. Do not explain or comment on the text content
14. **Do NOT translate**: Keep ALL content in English. Do NOT translate to any other language.

## Output Format

Must output JSON format containing a corrections array, each item being `{"id": integer, "text": "proofread text"}`.
- `id` maps one-to-one to the input line's `[id]`, and MUST cover every input id
- `text` holds ONLY the proofread content, without id, timestamp, or line-format tags

Return ONLY the proofread JSON, without any additional explanations or comments."""


def build_structured_calibrate_user_prompt(
input_data: dict = None,
dialogs_text: str = None,
Expand Down
23 changes: 19 additions & 4 deletions src/video_transcript_api/llm/segmenters/dialog_segmenter.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"""

import re
from typing import List, Dict, Any
from typing import List, Dict, Any, Optional

from ...utils.logging import setup_logger
from ..core.config import LLMConfig
Expand All @@ -15,16 +15,31 @@
class DialogSegmenter:
"""有说话人文本分段器"""

def __init__(self, config: LLMConfig):
def __init__(
self,
config: LLMConfig,
preferred_chunk_length: Optional[int] = None,
max_chunk_length: Optional[int] = None,
):
"""初始化对话分段器

Args:
config: LLM 配置
preferred_chunk_length: 理想块长覆盖(None 时取
config.preferred_chunk_length)。无说话人(plain 源)模式
用独立分块参数覆盖,has_speaker=True 路径不传、行为不变。
max_chunk_length: 最大块长覆盖(None 时取 config.max_chunk_length)
"""
self.config = config
self.min_chunk_length = config.min_chunk_length
self.max_chunk_length = config.max_chunk_length
self.preferred_chunk_length = config.preferred_chunk_length
self.max_chunk_length = (
max_chunk_length if max_chunk_length is not None else config.max_chunk_length
)
self.preferred_chunk_length = (
preferred_chunk_length
if preferred_chunk_length is not None
else config.preferred_chunk_length
)

def segment(self, dialogs: List[Dict]) -> List[List[Dict]]:
"""对对话列表进行智能分块
Expand Down
Loading
Loading