Skip to content

fix: preserve paragraph breaks as sentence boundaries in normalizer - #525

Merged
remsky merged 1 commit into
remsky:masterfrom
Christian-Sidak:fix/issue-519
Sep 6, 2026
Merged

fix: preserve paragraph breaks as sentence boundaries in normalizer#525
remsky merged 1 commit into
remsky:masterfrom
Christian-Sidak:fix/issue-519

Conversation

@Christian-Sidak

@Christian-Sidak Christian-Sidak commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The text normalizer was replacing all newlines (including \n\n paragraph breaks) with spaces, causing headings and body text to be merged into one continuous phrase with no prosodic pause.
  • Double-newline paragraph breaks are now converted to a period before the newline-to-space pass, giving the TTS engine a sentence boundary to pause on.
  • If the character preceding the paragraph break is already sentence-ending punctuation (., !, ?), no extra period is inserted, avoiding double punctuation.
  • Single newlines continue to become spaces as before.

Failure case from the issue:

Input:  "Steven Erikson\n\nFive riders drew rein in the pass."
Before: "Steven EriksonFive riders drew rein in the pass."  (merged, no pause)
After:  "Steven Erikson. Five riders drew rein in the pass."  (natural boundary)

Partial resolution of #519

Test plan

  • test_paragraph_breaks_preserved_as_sentence_boundary: exact failure case from the issue report
  • test_paragraph_break_no_double_punctuation: paragraph already ends with ., !, or ?
  • test_single_newline_becomes_space: soft line breaks still collapse to a space
  • test_multiple_blank_lines_treated_as_paragraph_break: three or more consecutive newlines produce exactly one period
  • All existing normalizer tests pass without modification

The normalizer replaced all newlines with spaces unconditionally,
merging paragraph-separated sections (headings, chapter titles, body
text) into one continuous phrase with no prosodic pause.

Convert double-newline paragraph breaks to a period before the
newline-to-space pass: if the preceding non-whitespace character is
already sentence-ending punctuation the period is omitted to avoid
duplication. Single newlines continue to become spaces.

Add tests covering the exact failure case from the issue report, no-
double-punctuation, single newline, and multiple consecutive blank
lines.

Fixes remsky#519

Signed-off-by: Christian-Sidak <61099993+Christian-Sidak@users.noreply.github.com>
@remsky

remsky commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Glad to merge in for the stated case fix, though I think to fully close #519 , the chunker needs to split at blank lines too for normalize: false to respect the paragraph breaks. The UAX change on PR #520 changed the flow a bit.

refactor/normalization-classes is a more general fix in the same line if you spot anything

@remsky
remsky merged commit fbbaa75 into remsky:master Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants